Information processing apparatus and non-transitory computer readable medium

ABSTRACT

An information processing apparatus and non-transitory computer readable medium and for processing information are provided. The information processing apparatus includes a processor configured to: receive image data; cause a display to display the received image data; and move, in response to a user instruction, a portion of the image data from a first place to a second place on the display. The portion of the image data is a partial region of an entire region of the image data. While the portion of the image data is being moved from the first place to the second place, the portion of the image data is displayed at the first place.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/987,058 filed Aug. 6, 2020, which is a continuation of U.S.application Ser. No. 16/665,507 filed Oct. 28, 2019, now U.S. Pat. No.10,764,452 issued on Sep. 1, 2020, which is a continuation of U.S.application Ser. No. 15/966,586, filed Apr. 30, 2018, now U.S. Pat. No.10,469,682 issued on Nov. 5, 2019, which is a continuation of U.S.patent application Ser. No. 15/349,134, filed Nov. 11, 2016, now U.S.Pat. No. 9,986,113 issued on May 29, 2018, which is based on and claimspriority under 35 USC 119 from Japanese Patent Applications No.2016-093290, No. 2016-093291, and No. 2016-093292 filed May 6, 2016, andJapanese Patent Application No. 2016-128350 filed Jun. 29, 2016, all ofwhich are incorporated herein by reference.

BACKGROUND Technical Field

The present invention relates to an information processing apparatus anda non-transitory computer readable medium.

SUMMARY

Aspects of non-limiting embodiments of the present disclosure relate todisplaying a first field of view of a camera directed to a firstdirection; capturing an image of a first target in the first direction;displaying the captured image of the first target; and while continuingto cause the display to display the captured image of the first target,causing the display to display, behind the captured image of the firsttarget, a second field of view of the camera directed to a seconddirection that is different from the first direction.

Aspects of certain non-limiting embodiments of the present disclosureovercome the above disadvantages and other disadvantages not describedabove. However, aspects of the non-limiting embodiments are not requiredto overcome the disadvantages described above, and aspects of thenon-limiting embodiments of the present disclosure may not overcome anyof the problems described above.

According to an aspect of the invention, there is provided anon-transitory computer readable medium storing a program that causes acomputer to execute information processing, the information processingcomprising: causing a display to display a first field of view of acamera directed to a first direction; capturing an image of a firsttarget in the first direction; causing the display to display thecaptured image of the first target; and while continuing to cause thedisplay to display the captured image of the first target, causing thedisplay to display, behind the captured image of the first target, asecond field of view of the camera directed to a second direction thatis different from the first direction.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a block diagram illustrating an image forming system accordingto a first exemplary embodiment of the present invention;

FIG. 2 is a block diagram illustrating an image forming apparatusaccording to the first exemplary embodiment;

FIG. 3 is a block diagram illustrating a server according to the firstexemplary embodiment;

FIG. 4 is a block diagram illustrating a terminal apparatus according tothe first exemplary embodiment;

FIG. 5 is a schematic diagram illustrating an appearance of the imageforming apparatus;

FIG. 6 is a diagram illustrating an example of a device functionmanagement table;

FIG. 7 is a diagram illustrating an example of a cooperative functionmanagement table;

FIG. 8 is a diagram illustrating an example of a function display screendisplayed on the terminal apparatus;

FIG. 9 is a diagram illustrating target devices that cooperate with eachother;

FIGS. 10A and 10B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the first exemplaryembodiment;

FIGS. 11A and 11B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the first exemplaryembodiment;

FIG. 12 is a diagram illustrating an example of a device functionmanagement table according to a first modification example;

FIGS. 13A and 13B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the first modificationexample;

FIGS. 14A and 14B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the first modificationexample;

FIGS. 15A and 15B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the first modificationexample;

FIGS. 16A to 16C are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a second modificationexample;

FIGS. 17A to 17C are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the second modificationexample;

FIG. 18 is a diagram illustrating an example of a screen displayed onthe terminal apparatus according to the second modification example;

FIGS. 19A and 19B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a third modificationexample;

FIGS. 20A and 20B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a fourth modificationexample;

FIG. 21 is a sequence diagram illustrating a connection processperformed in the image forming system according to the first exemplaryembodiment;

FIG. 22 is a block diagram illustrating a server according to a secondexemplary embodiment;

FIG. 23 is a diagram illustrating an example of a cooperative functionmanagement table;

FIGS. 24A to 24C are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the second exemplaryembodiment;

FIGS. 25A and 25B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the second exemplaryembodiment;

FIGS. 26A to 26C are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a fifth modificationexample;

FIG. 27 is a diagram illustrating an example of a screen displayed onthe terminal apparatus according to the fifth modification example;

FIG. 28 is a diagram illustrating an example of a screen displayed onthe terminal apparatus according to the fifth modification example;

FIGS. 29A and 29B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a sixth modificationexample;

FIGS. 30A and 30B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the sixth modificationexample;

FIGS. 31A to 31C are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to the sixth modificationexample;

FIGS. 32A and 32B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a seventh modificationexample;

FIGS. 33A and 33B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to an eighth modificationexample;

FIGS. 34A and 34B are diagrams illustrating an example of a screendisplayed on the terminal apparatus according to a tenth modificationexample; and

FIG. 35 is a diagram illustrating an example of a screen displayed onthe terminal apparatus.

DETAILED DESCRIPTION First Exemplary Embodiment

An image forming system serving as an information processing systemaccording to a first exemplary embodiment of the present invention willbe described with reference to FIG. 1. FIG. 1 illustrates an example ofthe image forming system according to the first exemplary embodiment.

The image forming system according to the first exemplary embodimentincludes an image forming apparatus 10, which is an example of a device;a terminal apparatus 12, which is an example of a device; a server 14;and a terminal apparatus 16, which is an example of an informationprocessing apparatus. The image forming apparatus 10, the terminalapparatuses 12 and 16, and the server 14 are connected to each otherthrough a communication path N such as a network. Of course, the imageforming apparatus 10, the terminal apparatuses 12 and 16, and the server14 may be connected to other apparatuses through different communicationpaths. In the example illustrated in FIG. 1, two devices (the imageforming apparatus 10 and the terminal apparatus 12) are included in theimage forming system. The image forming system may include one or moredevices. Also, the image forming system may include plural servers 14and plural terminal apparatuses 16.

The image forming apparatus 10 is an apparatus that has an image formingfunction. Specifically, the image forming apparatus 10 is an apparatusthat has at least one of a scan function, a print function, a copyfunction, and a facsimile function. The image forming apparatus 10 alsohas a function of transmitting data to and receiving data from anotherapparatus.

Each of the terminal apparatuses 12 and 16 is an apparatus such as apersonal computer (PC), a tablet PC, a smartphone, or a mobile phone,and has a function of transmitting data to and receiving data fromanother apparatus. The terminal apparatus 16 functions as, for example,a user interface unit (UI unit) when a device such as the image formingapparatus 10 is used.

The server 14 is an apparatus that manages the functions of individualdevices. For example, the server 14 manages cooperative functionsexecuted through cooperation between plural functions. The server 14also has a function of transmitting data to and receiving data fromanother apparatus.

In the example illustrated in FIG. 1, the image forming apparatus 10 andthe terminal apparatus 12 are included, as an example of devices, in theimage forming system, but this is merely an example. Other apparatuses,for example, a display apparatus such as a projector, a telephone, awatch, and a monitoring camera may be included as devices in the imageforming system. Of course, the image forming apparatus 10 and theterminal apparatus 12 are not necessarily included in the image formingsystem.

The server 14 may manage, for each user, functions available to theuser. For example, a function purchased by a user is a functionavailable to the user, and the server 14 may manage a function purchasehistory for each user. Of course, the server 14 may manage not onlyfunctions that are purchased or not purchased but also functions thatare available free of charge, additional updater functions, and specialfunctions managed by a manager. A function purchase process is performedby, for example, the server 14. Of course, the function purchase processmay be performed by another apparatus.

In the image forming system according to the first exemplary embodiment,plural cooperative functions to be executed through cooperation betweenplural functions are specified, and information about the pluralcooperative functions is displayed in accordance with an order ofpriority.

Hereinafter, the individual apparatuses included in the image formingsystem according to the first exemplary embodiment will be described indetail.

First, the configuration of the image forming apparatus 10 will bedescribed in detail with reference to FIG. 2. FIG. 2 illustrates theconfiguration of the image forming apparatus 10.

A communication unit 18 is a communication interface and has a functionof transmitting data to another apparatus and a function of receivingdata from another apparatus. The communication unit 18 may be acommunication interface having a wireless communication function or maybe a communication interface having a wired communication function.

An image forming unit 20 executes a function regarding image formation.Specifically, the image forming unit 20 executes at least one of a scanfunction, a print function, a copy function, and a facsimile function.When the scan function is executed, a document is read and scan data(image data) is generated. When the print function is executed, an imageis printed on a recording medium such as paper. When the copy functionis executed, a document is read and printed on a recording medium. Whenthe facsimile function is executed, image data is transmitted orreceived by facsimile. Furthermore, a cooperative function includingplural functions may be executed. For example, a scan and transferfunction, which is a combination of a scan function and a transmission(transfer) function, may be executed. When the scan and transferfunction is executed, a document is read, scan data (image data) isgenerated, and the scan data is transmitted to a destination (forexample, an external apparatus such as the terminal apparatus 16). Ofcourse, this cooperative function is merely an example and anothercooperative function may be executed.

A memory 22 is a storage apparatus such as a hard disk or a solid statedrive (SSD). The memory 22 stores, for example, information representingan image formation instruction (for example, job information), imagedata to be printed, scan data generated by executing a scan function,device address information representing the address of another device,server address information representing the address of the server 14,various pieces of control data, and various programs. Of course, thesepieces of information and data may be stored in different storageapparatuses or in one storage apparatus.

A UI unit 24 is a user interface unit and includes a display and anoperation unit. The display is a display apparatus such as a liquidcrystal display. The operation unit is an input apparatus such as atouch screen or a keyboard. The image forming apparatus 10 does notnecessarily include the UI unit 24, and may include a hardware userinterface unit (hardware UI unit) serving as hardware instead of thedisplay. The hardware UI unit is, for example, a hardware keypaddedicated to input numbers (for example, a numeric keypad) or a hardwarekeypad dedicated to indicate directions (for example, a directionindication keypad).

A controller 26 controls the operations of the individual units of theimage forming apparatus 10.

Next, the configuration of the server 14 will be described in detailwith reference to FIG. 3. FIG. 3 illustrates the configuration of theserver 14.

A communication unit 28 is a communication interface and has a functionof transmitting data to another apparatus and a function of receivingdata from another apparatus. The communication unit 28 may be acommunication interface having a wireless communication function or maybe a communication interface having a wired communication function.

A memory 30 is a storage apparatus such as a hard disk or an SSD. Thememory 30 stores, for example, device function information 32,cooperative function information 34, various pieces of data, variousprograms, device address information representing the addresses ofindividual devices, and server address information representing theaddress of the server 14. Of course, these pieces of information anddata may be stored in different storage apparatuses or in one storageapparatus. The device function information 32 and the cooperativefunction information 34 stored in the memory 30 may be provided to theterminal apparatus 16 periodically or at designated timing, andaccordingly the information stored in the terminal apparatus 16 may beupdated. Hereinafter, the device function information 32 and thecooperative function information 34 will be described.

The device function information 32 is information representing thefunctions of the individual devices (for example, the image formingapparatus 10 and the terminal apparatus 12) included in the imageforming system. For example, the device function information 32 isinformation representing the correspondence between deviceidentification information for identifying a device and one or pluralpieces of function identification information for identifying one orplural functions of the device. The device identification informationincludes, for example, a device ID, the name of a device, informationrepresenting the type of a device, a model number, and positioninformation. The function identification information includes, forexample, a function ID and the name of a function. For example, if acertain image forming apparatus 10 has a scan function, a printfunction, a copy function, and a scan and transfer function, the deviceidentification information of the image forming apparatus 10 isassociated with function identification information representing thescan function, function identification information representing theprint function, function identification information representing thecopy function, and function identification information representing thescan and transfer function. The functions of the individual devices arespecified (identified) by referring to the device function information32.

The cooperative function information 34 is information representingcooperative functions, each being executed through cooperation betweenplural functions. One or plural cooperative functions are executedthrough cooperation between plural functions. Each cooperative functionmay be executed through cooperation between plural functions of onedevice (for example, the image forming apparatus 10 or the terminalapparatus 12) or may be executed through cooperation between pluralfunctions of plural devices (for example, the image forming apparatus 10and the terminal apparatus 12). A terminal apparatus that provides anoperation instruction (in this exemplary embodiment, the terminalapparatus 16) may be included in the devices to be identified, and afunction of the terminal apparatus may be used as part of a cooperativefunction.

The cooperative function information 34 is, for example, informationrepresenting the correspondence among information representing acombination of pieces of function identification information foridentifying individual functions that cooperate with each other toexecute one or plural cooperative functions, cooperative functionidentification information for identifying the one or plural cooperativefunctions, information representing an order in which the individualfunctions used for the one or plural cooperative functions are linked,and information representing an order of priority of display of the oneor plural cooperative functions. The cooperative function identificationinformation includes, for example, a cooperative function ID and thename of a cooperative function. The order of priority of display is anorder in which information about the one or plural cooperative functions(for example, cooperative function identification information) isdisplayed on the terminal apparatus 16.

In the case of causing plural devices to cooperate with each other, thecooperative function information 34 is information representing one orplural cooperative functions to be executed through cooperation betweenplural functions of the plural devices, and is information representingthe correspondence among information representing a combination ofpieces of device identification information for identifying theindividual devices that cooperate with each other to execute the one orplural cooperative functions, cooperative function identificationinformation, information representing an order in which the individualdevices used for the one or plural cooperative functions are linked, andinformation representing an order of priority of display of the one orplural cooperative functions.

A cooperative function may be a function executed through cooperationbetween plural different functions or may be a function executed throughcooperation between the same functions. A cooperative function may be afunction that is not available without cooperation. The function that isnot available without cooperation may be a function that becomesavailable by combining the same functions or different functions amongthe functions of target devices that cooperate with each other. Forexample, the cooperation between a device having a print function(printer) and a device having a scan function (scanner) implements acopy function as a cooperative function. That is, the cooperationbetween the print function and the scan function implements the copyfunction. In this case, the copy function is associated with thecombination of the print function and the scan function. In thecooperative function information 34, for example, the cooperativefunction identification information for identifying the copy function asa cooperative function is associated with the combination of the deviceidentification information for identifying the device having the printfunction and the device identification information for identifying thedevice having the scan function.

The memory 30 may store function purchase history information. Thefunction purchase history information is information representing thefunction purchase histories of individual users, that is, informationrepresenting the functions purchased by the individual users. Forexample, the function purchase history information is informationrepresenting the correspondence between user identification informationfor identifying a user and function identification informationrepresenting a function purchased by the user. The user identificationinformation is, for example, user account information such as a user IDand a user name. A function purchased by a user is a function availableto the user. Functions purchased by individual users, that is, functionsavailable to the individual users, are specified by referring to thefunction purchase history information. The function purchase historyinformation is updated every time a user purchases a function, forexample.

A controller 36 controls the operations of the individual units of theserver 14. The controller 36 includes a specifying unit 38.

The specifying unit 38 receives device identification information foridentifying a device and specifies one or plural pieces of functionidentification information of one or plural functions associated withthe device identification information in the device function information32 stored in the memory 30. Accordingly, the one or plural functions ofthe device are specified. For example, device identification informationis transmitted from the terminal apparatus 16 to the server 14, and thenthe specifying unit 38 specifies one or plural pieces of functionidentification information of one or plural functions associated withthe device identification information. Information about the one orplural functions (for example, function identification information andfunction explanation information) is transmitted from the server 14 tothe terminal apparatus 16 and is displayed on the terminal apparatus 16,for example. Accordingly, the information about the one or pluralfunctions of the device specified by the device identificationinformation is displayed on the terminal apparatus 16.

Also, the specifying unit 38 receives pieces of device identificationinformation for identifying target devices that cooperate with eachother and specifies one or plural pieces of cooperative functionidentification information of one or plural cooperative functionsassociated with the combination of the pieces of device identificationinformation in the cooperative function information 34 stored in thememory 30. Accordingly, the one or plural cooperative functions to beexecuted through cooperation between the functions of the target devicesthat cooperate with each other are specified. For example, plural piecesof device identification information are transmitted from the terminalapparatus 16 to the server 14, and then the specifying unit 38 specifiesone or plural pieces of cooperative function identification informationof one or plural cooperative functions associated with the plural piecesof device identification information. Information about the one orplural cooperative functions (for example, cooperative functionidentification information and cooperative function explanationinformation) is transmitted from the server 14 to the terminal apparatus16 and is displayed on the terminal apparatus 16, for example.Accordingly, the information about the one or plural cooperativefunctions to be executed by the plural devices identified by the pluralpieces of device identification information is displayed on the terminalapparatus 16.

Also, the specifying unit 38 refers to the cooperative functioninformation 34 and determines, on the basis of an order in which theindividual functions used for cooperative functions are linked, an orderof priority in which information about the cooperative functions isdisplayed. For example, plural cooperative functions may be implementedby causing plural functions to cooperate with each other. In this case,an order of priority in which information about the individualcooperative functions is displayed is determined on the basis of anorder in which the individual functions are linked. The informationabout the cooperative functions is displayed on the terminal apparatus16 in accordance with the order of priority. For example, images relatedto devices are displayed on the terminal apparatus 16, and the imagesare linked to each other by a user. In the case of causing pluralfunctions of plural devices to cooperate with each other, plural imagesrelated to the plural devices are linked to each other by the user. Theorder in which the images are linked corresponds to the linkage order ofthe functions. The specifying unit 38 specifies, on the basis of theorder in which the images are linked, an order of priority in whichinformation about the cooperative functions is displayed.

In a case where a function purchase history is managed, the specifyingunit 38 may receive user identification information for identifying auser and may specify the pieces of function identification informationof individual functions associated with the user identificationinformation in the function purchase history information stored in thememory 30. Accordingly, a group of functions purchased by the user, thatis, a group of functions available to the user, is specified(identified). For example, user identification information istransmitted from the terminal apparatus 16 to the server 14, and thepieces of function identification information of individual functionsassociated with the user identification information are specified by thespecifying unit 38. The pieces of function identification information ofthe individual functions (for example, pieces of informationrepresenting the names of the functions) are transmitted from the server14 to the terminal apparatus 16 and are displayed on the terminalapparatus 16, for example. Accordingly, the pieces of information foridentifying the functions available to the user specified by the useridentification information (function identification information) aredisplayed on the terminal apparatus 16. For example, the specifying unit38 receives device identification information and user identificationinformation, specifies one or plural pieces of function identificationinformation of one or plural functions associated with the deviceidentification information in the device function information 32, andalso specifies one or plural pieces of function identificationinformation of one or plural functions associated with the useridentification information in the function purchase history information.Accordingly, the one or plural functions that the device specified bythe device identification information has and that are available to theuser specified by the user identification information are specified. Theone or plural pieces of function identification information representingthe one or plural functions are transmitted from the server 14 to theterminal apparatus 16 and are displayed on the terminal apparatus 16,for example.

The controller 36 may execute a function purchase process and may managethe history of the purchase. For example, if a pay function is purchasedby a user, the controller 36 applies a charging process to the user andcreates function purchase history information representing the historyof the purchase.

Next, the configuration of the terminal apparatus 16 will be describedin detail with reference to FIG. 4. FIG. 4 illustrates the configurationof the terminal apparatus 16.

A communication unit 40 is a communication interface and has a functionof transmitting data to another apparatus and a function of receivingdata from another apparatus. The communication unit 40 may be acommunication interface having a wireless communication function or maybe a communication interface having a wired communication function.

A camera 42, which serves as an image capturing unit, captures an imageof a subject and thereby generates image data (for example, still imagedata or moving image data). Alternatively, instead of using the camera42 of the terminal apparatus 16, image data captured by an externalcamera connected to a communication path such as a network may bereceived by the communication unit 40 and may be displayed on a UI unit46 so that the image data may be operated.

A memory 44 is a storage apparatus such as a hard disk or an SSD. Thememory 44 stores various programs, various pieces of data, the addressinformation of the server 14, the pieces of address information ofindividual devices (for example, the pieces of address information ofthe image forming apparatus 10 and the terminal apparatus 12),information about identified target devices that cooperate with eachother, and information about cooperative functions.

The UI unit 46 is a user interface unit and includes a display and anoperation unit. The display is a display apparatus such as a liquidcrystal display. The operation unit is an input apparatus such as atouch screen, a keyboard, or a mouse. Of course, a user interface thatserves as both a display and an operation unit may be used (for example,a touch display or a display on which an electronic keyboard or the likeis displayed).

A controller 48 controls the operations of the individual units of theterminal apparatus 16. The controller 48 functions as, for example, adisplay controller (controller) and causes the display of the UI unit 46to display various pieces of information.

The display of the UI unit 46 displays, for example, an image capturedby the camera 42, an image related to a device identified as a targetdevice to be used (for example, a target device that cooperates), and soforth. An image related to a device may be an image (still image ormoving image) representing the device captured by the camera 42 or maybe an image (for example, an icon) schematically representing thedevice. The data of the image schematically representing the device maybe stored in the server 14 and provided from the server 14 to theterminal apparatus 16, or may be stored in the terminal apparatus 16 inadvance. For example, in the case of executing a cooperative functionthrough cooperation between plural devices, images of the plural devicesare captured by the camera 42 and the images of the devices aredisplayed on the display of the UI unit 46. When the user links theimages related to the target devices that cooperate with each other, oneor plural cooperative functions to be executed by the target devices arespecified, and information about the cooperative functions is displayedon the display of the UI unit 46 in accordance with an order ofpriority, which is based on the order of the linkage.

The terminal apparatus 12 serving as a device has, for example, aconfiguration similar to that of the terminal apparatus 16. The terminalapparatus 12 does not necessarily include the camera 42.

The above-described device function information 32 may be stored in thememory 44 of the terminal apparatus 16. In this case, the devicefunction information 32 is not necessarily stored in the memory 30 ofthe server 14. Also, the above-described cooperative functioninformation 34 may be stored in the memory 44 of the terminal apparatus16. In this case, the cooperative function information 34 is notnecessarily stored in the memory 30 of the server 14. The controller 48of the terminal apparatus 16 may include the above-described specifyingunit 38, which may specify a function of a device by identifying thedevice on the basis of device identification information and may specifya cooperative function to be executed through cooperation between pluralfunctions. In this case, the server 14 does not necessarily include thespecifying unit 38. In a case where function purchase historyinformation is created, the function purchase history information may bestored in the memory 44 of the terminal apparatus 16. In this case, thefunction purchase history information is not necessarily stored in thememory 30 of the server 14. The controller 48 of the terminal apparatus16 may manage a function purchase history of a user. In this case, thecontroller 36 of the server 14 does not necessarily have the managementfunction therefor. The controller 48 of the terminal apparatus 16 mayspecify a function available to a user on the basis of useridentification information. Alternatively, the device functioninformation 32 and the cooperative function information 34 may be storedin the devices such as the image forming apparatus 10 and the terminalapparatus 12, and the devices such as the image forming apparatus 10 andthe terminal apparatus 12 may include the specifying unit 38. That is,the process by the specifying unit 38 of the server 14 may be performedin the server 14, may be performed in the terminal apparatus 16, or maybe performed in the devices such as the image forming apparatus 10 andthe terminal apparatus 12.

In the first exemplary embodiment, for example, augmented reality (AR)technologies are applied to obtain device identification information andto identify a target device to be used. For example, AR technologies areapplied to obtain pieces of device identification information of thetarget devices that cooperate with each other and to identify the targetdevices. The AR technologies according to the related art are used. Forexample, a marker-based AR technology in which a marker such as atwo-dimensional barcode is used, a markerless AR technology in which animage recognition technique is used, a position information ARtechnology in which position information is used, and the like are used.Of course, device identification information may be obtained and atarget device to be used may be identified without applying the ARtechnologies. For example, in the case of a device connected to anetwork, the device may be identified on the basis of the IP addressthereof or by reading the device ID thereof.

Hereinafter, a process of obtaining the device identificationinformation of a device will be described in detail with reference toFIG. 5. As an example, a description will be given of the case ofobtaining the device identification information of the image formingapparatus 10. FIG. 5 schematically illustrates an appearance of theimage forming apparatus 10. Here, a description will be given of aprocess of obtaining the device identification information by applyingthe marker-based AR technology. The housing of the image formingapparatus 10 is attached with a marker 50, such as a two-dimensionalbarcode. The marker 50 is information obtained by coding the deviceidentification information of the image forming apparatus 10. The useractivates the camera 42 of the terminal apparatus 16 and captures, withthe camera 42, an image of the marker 50 attached to the image formingapparatus 10, which is to be used. Accordingly, image data representingthe marker 50 is generated. The image data is transmitted, for example,from the terminal apparatus 16 to the server 14. In the server 14, thecontroller 36 performs a decoding process on the marker imagerepresented by the image data and thereby extracts device identificationinformation. Accordingly, the image forming apparatus 10 to be used (theimage forming apparatus 10 attached with the marker 50 whose image hasbeen captured) is identified. The specifying unit 38 of the server 14specifies the pieces of function identification information of functionsassociated with the extracted device identification information in thedevice function information 32. Accordingly, the functions of the imageforming apparatus 10 to be used are specified.

Alternatively, the controller 48 of the terminal apparatus 16 mayperform a decoding process on the image data representing the marker 50to extract the device identification information. In this case, theextracted device identification information is transmitted from theterminal apparatus 16 to the server 14. The specifying unit 38 of theserver 14 specifies the pieces of function identification information offunctions associated with the device identification informationtransmitted from the terminal apparatus 16 in the device functioninformation 32. In a case where the device function information 32 isstored in the memory 44 of the terminal apparatus 16, the controller 48of the terminal apparatus 16 may specify the pieces of functionidentification information of functions associated with the extracteddevice identification information in the device function information 32.

The marker 50 may include coded function identification information ofthe functions of the image forming apparatus 10. In this case, thedevice identification information of the image forming apparatus 10 isextracted and also the pieces of function identification information ofthe functions of the image forming apparatus 10 are extracted byperforming a decoding process on the image data representing the marker50. Accordingly, the image forming apparatus 10 is specified and alsothe functions of the image forming apparatus 10 are specified. Thedecoding process may be performed by the server 14 or the terminalapparatus 16.

In the case of executing a cooperative function through cooperationbetween plural devices, images of the markers of the devices arecaptured to obtain the pieces of device identification information ofthe devices, and thereby the cooperative function is specified.

In the case of obtaining device identification information by applyingthe markerless AR technology, for example, the user captures an image ofthe whole appearance or part of the appearance of the device to be used(for example, the image forming apparatus 10) by using the camera 42 ofthe terminal apparatus 16. Of course, it is useful to obtain informationfor specifying the device to be used, such as the name (for example, thetrade name) or model number of the device, by capturing an image of theappearance of the device. As a result of the capturing, appearance imagedata representing the whole appearance or part of the appearance of thedevice to be used is generated. The appearance image data istransmitted, for example, from the terminal apparatus 16 to the server14. In the server 14, the controller 36 identifies the device to be usedon the basis of the appearance image data. For example, the memory 30 ofthe server 14 stores, for each device, appearance image correspondenceinformation representing the correspondence between appearance imagedata representing the whole appearance or part of the appearance of thedevice and device identification information of the device. Thecontroller 36 compares, for example, the appearance image data receivedfrom the terminal apparatus 16 with each piece of appearance image dataincluded in the appearance image correspondence information, andspecifies the device identification information of the device to be usedon the basis of the comparison result. For example, the controller 36extracts, from the appearance image data received from the terminalapparatus 16, a feature of the appearance of the device to be used,specifies the appearance image data representing a feature that is thesame as or similar to the feature of the appearance in the appearanceimage data group included in the appearance image correspondenceinformation, and specifies the device identification informationassociated with the appearance image data. Accordingly, the device to beused (the device whose image has been captured by the camera 42) isrecognized. Alternatively, in a case where an image showing the name(for example, the trade name) or model number of the device is capturedand appearance image data representing the name or model number isgenerated, the device to be used may be identified on the basis of thename or model number represented by the appearance image data. Thespecifying unit 38 of the server 14 specifies the pieces of functionidentification information of the individual functions associated withthe specified device identification information in the device functioninformation 32. Accordingly, the functions of the device to be used (forexample, the image forming apparatus 10) are specified.

Alternatively, the controller 48 of the terminal apparatus 16 maycompare the appearance image data representing the whole appearance orpart of the appearance of the device to be used (for example, the imageforming apparatus 10) with each piece of appearance image data includedin the appearance image correspondence information and may specify thedevice identification information of the device to be used on the basisof the comparison result. The appearance image correspondenceinformation may be stored in the memory 44 of the terminal apparatus 16.In this case, the controller 48 of the terminal apparatus 16 refers tothe appearance image correspondence information stored in the memory 44of the terminal apparatus 16 and thereby specifies the deviceidentification information of the device to be used. Alternatively, thecontroller 48 of the terminal apparatus 16 may obtain the appearanceimage correspondence information from the server 14 and may refer to theappearance image correspondence information, so as to specify the deviceidentification information of the device to be used.

In the case of executing a cooperative function through cooperationbetween plural devices, the pieces of device identification informationof the individual devices are obtained by capturing images of the wholeappearance or part of the appearance of the individual devices thatcooperate with each other, and thereby the cooperative function isspecified.

In the case of obtaining device identification information by applyingthe position information AR technology, for example, positioninformation representing the position of a device (for example, theimage forming apparatus 10) is obtained by using a global positioningsystem (GPS) function. For example, each device has a GPS function andobtains device position information representing the position of thedevice. The terminal apparatus 16 outputs, to the device to be used,information representing a request for obtaining device positioninformation, and receives, as a response to the request, the deviceposition information of the device from the device. The device positioninformation is, for example, transmitted from the terminal apparatus 16to the server 14. In the server 14, the controller 36 identifies thedevice to be used on the basis of the device position information. Forexample, the memory 30 of the server 14 stores, for each device,position correspondence information representing the correspondencebetween the device position information representing the position of thedevice and the device identification information of the device. Thecontroller 36 specifies, in the position correspondence information, thedevice identification information associated with the device positioninformation received from the terminal apparatus 16. Accordingly, thedevice to be used is recognized. The specifying unit 38 of the server 14specifies, in the device function information 32, the pieces of functionidentification information of the individual functions associated withthe specified device identification information. Accordingly, thefunctions of the device to be used (for example, the image formingapparatus 10) are recognized.

The controller 48 of the terminal apparatus 16 may specify, in theposition correspondence information, the device identificationinformation associated with the position information of the device to beused. The position correspondence information may be stored in thememory 44 of the terminal apparatus 16. In this case, the controller 48of the terminal apparatus 16 refers to the position correspondenceinformation stored in the memory 44 of the terminal apparatus 16 andthereby specifies the device identification information of the device tobe used. Alternatively, the controller 48 of the terminal apparatus 16may obtain the position correspondence information from the server 14and refer to the position correspondence information, so as to specifythe device identification information of the device to be used.

In the case of executing a cooperative function through cooperationbetween plural devices, the pieces of device position information of thedevices are obtained and the pieces of device identification informationof the devices are specified on the basis of the pieces of deviceposition information. Accordingly, the cooperative function isspecified.

Hereinafter, the image forming system according to the first exemplaryembodiment will be described in detail.

First, the device function information 32 will be described in detailwith reference to FIG. 6. FIG. 6 illustrates an example of a devicefunction management table, which is the device function information 32.In the device function management table, for example, a device ID,information representing the name of a device (for example, the type ofa device), information representing one or plural functions of thedevice (function identification information), and an image ID areassociated with each other. The device ID and the name of a devicecorrespond to an example of device identification information. The imageID is an example of image identification information for identifying animage schematically representing a device (for example, an icon). Thedevice function management table does not necessarily include the imageID. For example, the device having a device ID “B” is a multifunctionperipheral (MFP, an image forming apparatus having plural image formingfunctions) and has a print function, a scan function, and so forth. Animage ID for identifying an image schematically representing the deviceis associated with the device. The data of the image schematicallyrepresenting the device is stored in, for example, the memory 30 of theserver 14.

For example, with use of the AR technologies, a device ID foridentifying a device to be used is obtained. The specifying unit 38 ofthe server 14 specifies the name of the device, one or plural functions,and image ID associated with the device ID by referring to the devicefunction management table. Accordingly, the device to be used isidentified. For example, information representing the name of the deviceand the data of an image schematically representing the device aretransmitted from the server 14 to the terminal apparatus 16 and thenthey are displayed on the UI unit 46 of the terminal apparatus 16. Theimage schematically representing the device is displayed as an imagerelated to the device. Of course, instead of the image schematicallyrepresenting the device, an image captured by the camera 42 may bedisplayed on the UI unit 46 of the terminal apparatus 16. If the imagerelated to the device (for example, the image captured by the camera 42or the image schematically representing the device) is designated by theuser on the UI unit 46 of the terminal apparatus 16, information aboutthe functions of the device (for example, function identificationinformation or function explanation information) may be transmitted fromthe server 14 to the terminal apparatus 16 and may be displayed on theUI unit 46 of the terminal apparatus 16.

Next, the cooperative function information 34 will be described indetail with reference to FIG. 7. FIG. 7 illustrates an example of acooperative function management table, which is the cooperative functioninformation 34. In the cooperative function management table, forexample, information representing a combination of device IDs,information representing the names (types) of target devices thatcooperate with each other, information representing one or pluralcooperative functions (cooperative function identification information),information representing a linkage order, and information representingan order of priority are associated with each other. The linkage ordercorresponds to an order in which images related to devices are linked.The order of priority is an order of priority in which information aboutcooperative functions is displayed. For example, the device having adevice ID “A” is a personal computer (PC), and the device having adevice ID “B” is an MFP. The cooperation between the PC (A) and the MFP(B) implements, for example, a scan and transfer function and a printfunction as cooperative functions. The scan and transfer function is afunction of transferring image data generated through scanning by theMFP (B) to the PC (A). The print function is a function of transmittingdata (for example, image data or document data) stored in the PC (A) tothe MFP (B) and printing the data by the MFP (B). For example, if a linkis made from the MFP (B) to the PC (A), that is, if a link is made froman image related to the MFP (B) to an image related to the PC (A), theorder of priority of the scan and transfer function is “1” and the orderof priority of the print function is “2”. In this case, informationabout the scan and transfer function is displayed with priority overinformation about the print function. On the other hand, if a link ismade from the PC (A) to the MFP (B), that is, if a link is made from animage related to the PC (A) to an image related to the MFP (B), theorder of priority of the print function is “1” and the order of priorityof the scan and transfer function is “2”. In this case, informationabout the print function is displayed with priority over informationabout the scan and transfer function.

Next, a screen displayed on the UI unit 46 of the terminal apparatus 16will be described in detail with reference to FIG. 8. FIG. 8 illustratesan example of a function display screen. The function display screen isa screen that is displayed on the UI unit 46 of the terminal apparatus16 in the case of using a single device.

For example, with use of any of the above-described marker-based ARtechnology, markerless AR technology, and position information ARtechnology, the device identification information of the device to beused (for example, the image forming apparatus 10) is obtained and thedevice is identified, and the pieces of function identificationinformation representing the individual functions associated with thedevice identification information, that is, the pieces of functionidentification information representing the individual functions of thedevice to be used (identified device) are specified. Alternatively, thepieces of function identification information representing theindividual functions associated with the user identification informationof the user who uses the target device, that is, the pieces of functionidentification information representing the individual functionsavailable to the user, may be specified. These pieces of information aredisplayed, as a function display screen, on the UI unit 46 of theterminal apparatus 16. Alternatively, since the functions of the deviceto be used are specified, functions that the device to be used does nothave among the functions in sale may be specified. The pieces offunction identification information representing the individualfunctions that the device to be used does not have may be displayed onthe function display screen.

On a function display screen 52 illustrated in FIG. 8, a button image 54representing function A, a button image 56 representing function B, anda button image 58 representing function C are displayed as an example ofpieces of function identification information. Function A is a functionof the device to be used (the device identified by using the ARtechnologies or the like) and is a function available to the user, thatis, a function purchased by the user. Function B is a function of thedevice to be used and is a function unavailable to the user, that is, afunction not purchased by the user. The user becomes able to usefunction B by purchasing it. Function C is a function that the device tobe used does not have, that is, a function incompatible with the deviceto be used. In accordance with whether or not the function representedby a button image is a function of the device to be used, the controller48 of the terminal apparatus 16 may change the display form of thebutton image. Also, in accordance with whether or not the functionrepresented by a button image is a function available to the user, thecontroller 48 may change the display form of the button image. Forexample, the controller 48 may change the color or shape of each buttonimage. In the example illustrated in FIG. 8, the button images 54, 56,and 58 are displayed in different colors. For example, a button imagerepresenting a function that the device to be used has and that isavailable to the user (for example, the button image 54 representingfunction A) is displayed in blue. A button image representing a functionthat the device to be used has and that is unavailable to the user (forexample, the button image 56 representing function B) is displayed inyellow. A button image representing a function that the device to beused does not have (for example, the button image 58 representingfunction C) is displayed in gray. Alternatively, the controller 48 maychange the shapes of the button images 54, 56, and 58, or may change thefonts of the function display names. Of course, the display form may bechanged in another method. Accordingly, the user may be able torecognize the availability of each function with enhanced visibility.

For example, if the user designates the button image 54 representingfunction A by using the terminal apparatus 16 and provides aninstruction to execute function A, execution instruction informationrepresenting an instruction to execute function A is transmitted fromthe terminal apparatus 16 to the target device. The executioninstruction information includes control data for executing function A,image data to be subjected to the process by function A, and so forth.In response to receipt of the execution instruction information, thetarget device executes function A in accordance with the executioninstruction information. For example, if the target device is the imageforming apparatus 10 and if function A is a scan and transfer function,the image forming unit 20 of the image forming apparatus 10 executes ascan function to generate scan data (image data). The scan data is thentransmitted from the image forming apparatus 10 to a destination that isset (for example, the terminal apparatus 16).

If the user designates the button image 56 representing function B byusing the terminal apparatus 16 and provides an instruction to purchasefunction B, a purchase process is performed. If the purchase process isperformed by the server 14, the terminal apparatus 16 accesses theserver 14. Accordingly, a screen for purchasing function B (for example,a website), which is information enabling the user to use function B, isdisplayed on the UI unit 46 of the terminal apparatus 16. By taking apurchase procedure on the screen, the user is allowed to use function B.For example, the terminal apparatus 16 stores a program of a webbrowser. With use of the web browser, the user is able to access theserver 14 from the terminal apparatus 16. When the user accesses theserver 14 by using the web browser, a function purchase screen (forexample, a website) is displayed on the UI unit 46 of the terminalapparatus 16, and the user is able to purchase the function through thewebsite. Of course, the purchase process may be performed by a serverdifferent from the server 14. Alternatively, as the information enablingthe user to use function B, a usage permission request screen (forexample, a website) for requesting the use of function B to a manager orthe like may be displayed on the UI unit 46 of the terminal apparatus16. If the user requests permission to use function B to the manager orthe like through the usage permission request screen and if thepermission is obtained, the user is able to use function B.

Hereinafter, a detailed description will be given of a method forcausing plural functions to cooperate with each other by applying the ARtechnologies.

With reference to FIG. 9, a description will be given of a method foridentifying plural devices by applying the marker-based AR technology orthe markerless AR technology and causing plural functions of the pluraldevices to cooperate with each other. FIG. 9 illustrates an example oftarget devices that cooperate with each other. For example, it isassumed that the image forming apparatus 10 and the terminal apparatus12 are used as the target devices. The image forming apparatus 10 is amultifunction peripheral (MFP) and the terminal apparatus 12 is apersonal computer (PC), for example. The image forming apparatus 10 andthe terminal apparatus 12 are devices that exist in real space. Theterminal apparatus 16 is an apparatus that exists in real space and thatis used to cause plural functions to cooperate with each other. Theterminal apparatus 16 is a mobile terminal apparatus such as asmartphone or a mobile phone, for example.

For example, the marker 50, such as a two-dimensional barcode, isattached to the housing of the image forming apparatus 10, and a marker60, such as a two-dimensional barcode, is attached to the housing of theterminal apparatus 12 (PC). The marker 60 is information obtained bycoding the device identification information of the terminal apparatus12. In a case where the marker-based AR technology or the markerless ARtechnology is used, the user captures, by using the camera 42 of theterminal apparatus 16 (for example, a smartphone), an image of the imageforming apparatus 10 and the terminal apparatus 12 as the target devicesthat cooperate with each other. In the example illustrated in FIG. 9, animage of both the image forming apparatus 10 and the terminal apparatus12 is captured in a state where both the image forming apparatus 10 andthe terminal apparatus 12 are within the field of view of the camera 42.Accordingly, image data representing the markers 50 and 60 is generatedand the image data is transmitted from the terminal apparatus 16 to theserver 14. In the server 14, the controller 36 performs a decodingprocess on the image data to extract the device identificationinformation of the image forming apparatus 10 and the deviceidentification information of the terminal apparatus 12, and therebyidentifies the image forming apparatus 10 and the terminal apparatus 12.Alternatively, appearance image data representing the appearances ofboth the image forming apparatus 10 and the terminal apparatus 12 may begenerated and the appearance image data may be transmitted from theterminal apparatus 16 to the server 14. In this case, in the server 14,the controller 36 specifies the device identification information of theimage forming apparatus 10 and the device identification information ofthe terminal apparatus 12 by referring to the appearance imagecorrespondence information. Accordingly, the image forming apparatus 10and the terminal apparatus 12 are identified. Of course, the process foridentifying the target devices that cooperate with each other may beperformed by the terminal apparatus 16.

A device display screen 62 is displayed on the display of the UI unit 46of the terminal apparatus 16, and a device image 64 related to the imageforming apparatus 10 and a device image 66 related to the terminalapparatus 12 are displayed on the device display screen 62. The deviceimages 64 and 66 may be images (having an original size at the capturingor an increased or decreased size) of the devices (the image formingapparatus 10 and the terminal apparatus 12) captured by the camera 42,or may be pieces of image data that are related to the identifieddevices and that are prepared in advance (not images obtained throughcapturing but schematic images (for example, icons)). As describedabove, images representing the devices captured by an external cameramay be used.

For example, in the case of using image data obtained by capturing animage of a device, the appearance of the device in a current state (forexample, an appearance including a scratch, note, sticker attached tothe device, and so forth) is reflected in the image, and thus the usermay be able to visually recognize the difference from another device ofthe same type more clearly.

In the case of using a schematic image, the data of the schematic imageis transmitted, for example, from the server 14 to the terminalapparatus 16. For example, when the image forming apparatus 10 isidentified, the specifying unit 38 of the server 14 specifies theschematic image related to the image forming apparatus 10 by referringto the device function management table (device function information 32)illustrated in FIG. 6. The data of the schematic image is transmittedfrom the server 14 to the terminal apparatus 16, and the schematic imageis displayed as the device image 64 on the device display screen 62. Thesame applies to the device image 66 related to the terminal apparatus12. The data of the device images 64 and 66 as schematic images may bestored in the terminal apparatus 16 in advance. In this case, when theimage forming apparatus 10 is identified, the device image 64 stored inthe terminal apparatus 16 is displayed on the device display screen 62.The same applies to the terminal apparatus 12. The data of schematicimages may be stored in an apparatus other than the server 14 and theterminal apparatus 16.

When a device is identified, information representing the name of thedevice may be transmitted from the server 14 to the terminal apparatus16, and the name of the device may be displayed on the device displayscreen 62 of the terminal apparatus 16. In the example illustrated inFIG. 9, the image forming apparatus 10 is an MFP, and the name thereofis “MFP (B)”. The terminal apparatus 12 is a PC, and the name thereof is“PC (A)”.

If a device as a target of image capturing is not identified even if theAR technologies or the like are applied, the device image representingthe target device is not necessarily displayed on the device displayscreen 62. Accordingly, the visibility of an identified device mayincrease. For example, if there are an identified device and anunidentified device and if images of the individual devices are capturedby the camera 42, the device image representing the unidentified deviceis not displayed. Accordingly, the device image representing theidentified device is displayed while being distinguished from the deviceimage representing the unidentified device, and thus the visibility ofthe identified device may increase. Alternatively, the device imagerepresenting the identified device (for example, the device images 64and 66) may be displayed in a highlighted manner. For example, thedevice image representing the identified device may be displayed in aspecific color, may be displayed by highlighting the edge of the deviceimage, may be displayed by enlarging the device image, may be displayedthree-dimensionally, or may be displayed by blinking the device image.Accordingly, the visibility of the identified device may increase.

With the user's operation of the device images 64 and 66 on the devicedisplay screen 62, the image forming apparatus 10 and the terminalapparatus 12 are designated as the target devices that cooperate witheach other, and the cooperative functions to be executed throughcooperation between the image forming apparatus 10 and the terminalapparatus 12 are specified. The information about the cooperativefunctions is displayed on the UI unit 46 of the terminal apparatus 16 inaccordance with the order of priority corresponding to the order inwhich the device images are operated.

Hereinafter, an operation for specifying a cooperative function will bedescribed in detail with reference to FIGS. 10A to 11B.

FIGS. 10A to 11B illustrate an example of a screen displayed on the UIunit 46 of the terminal apparatus 16. For example, it is assumed thatthe image forming apparatus 10 and the terminal apparatus 12 areidentified as the target devices that cooperate with each other. Asillustrated in FIG. 10A, the device display screen 62 is displayed onthe UI unit 46 of the terminal apparatus 16, and the device images 64and 66 are displayed on the device display screen 62. In this state, theuser links the device images representing the target devices to eachother by using an indicator (for example, a user's finger, a pen, or astylus). The controller 48 of the terminal apparatus 16 detects thetouch of the indicator on the device display screen 62 and detects amovement of the indicator on the device display screen 62. For example,as indicated by an arrow 68, the user touches the device image 64 on thedevice display screen 62 by using an operator and moves the operator tothe device image 66 on the device display screen 62, and thereby linksthe device image 64 to the device image 66. Accordingly, the imageforming apparatus 10 (MFP (B)) related to the device image 64 and theterminal apparatus 12 (PC (A)) related to the device image 66 aredesignated as the target devices that cooperate with each other and alinkage order is designated. The order in which the device images arelinked corresponds to the linkage order. In the example illustrated inFIG. 10A, a link is made from the device image 64 to the device image66, that is, from the image forming apparatus 10 (MFP (B)) to theterminal apparatus 12 (PC (A)). The information representing the linkageorder of the devices is transmitted from the terminal apparatus 16 tothe server 14. The controller 48 of the terminal apparatus 16 may causean image representing the trail of the movement of the operatorperformed by the user to be displayed on the device display screen 62.After the devices are linked to each other, the controller 48 of theterminal apparatus 16 may cause a preset straight line or the likerepresenting the trail to be displayed on the device display screen 62.

When the target devices that cooperate with each other (for example, theimage forming apparatus 10 and the terminal apparatus 12) are designatedin the above-described manner, the specifying unit 38 of the server 14specifies the cooperative functions associated with the combination ofthe PC (A) and the MFP (B) in the cooperative function management table(cooperative function information 34) illustrated in FIG. 7.Accordingly, the cooperative functions to be executed throughcooperation between the PC (A) and the MFP (B) are specified. When thelinkage order of the devices is designated by the user, the specifyingunit 38 specifies the order of priority associated with the linkageorder in the cooperative function management table. Specifically, withreference to FIG. 7, since the PC (A) and the MFP (B) are designated asthe target devices that cooperate with each other, the cooperativefunctions to be executed by these devices are a scan and transferfunction and a print function. Furthermore, since the link is made fromthe MFP (B) to the PC (A) (B→A), the order of priority of the scan andtransfer function is “1” and the order of priority of the print functionis “2”.

The information about the specified cooperative functions and theinformation about the specified order of priority are transmitted fromthe server 14 to the terminal apparatus 16. The controller 48 of theterminal apparatus 16 causes the UI unit 46 to display the informationabout the cooperative functions as the information about candidatecooperative functions in accordance with the order of priority.

For example, as illustrated in FIG. 10B, the controller 48 of theterminal apparatus 16 causes the UI unit 46 to display a cooperativefunction display screen 70 and to display the information about thecandidate cooperative functions on the cooperative function displayscreen 70. Since the order of priority of the scan and transfer functionis “1” and the order of priority of the print function is “2”, theinformation about the scan and transfer function is displayed withpriority over (for example, above) the information about the printfunction. For example, as the information about the scan and transferfunction, an explanation of the scan and transfer function “transfer thedata scanned by MFP (B) to PC (A)” is displayed. Also, as theinformation about the print function, an explanation of the printfunction “print the data in PC (A)” is displayed.

If a cooperative function is designated by the user and an executioninstruction is provided, the designated cooperative function isexecuted. For example, if a “YES” button is pressed by the user, thecooperative function related to the “YES” button is executed. Also, a“back” button is displayed on the cooperative function display screen70. If the “back” button is pressed by the user, the process ofconnecting the devices is stopped.

The process of specifying cooperative functions and the process ofspecifying an order of priority may be performed by the terminalapparatus 16.

The target devices that cooperate with each other may be designated andthe linkage order thereof may be designated by drawing circles aroundthe device images, instead of moving the operator between device images.For example, the order of a drawing operation corresponds to the linkageorder. Alternatively, the target devices that cooperate with each otherand the linkage order thereof may be designated in accordance with avoice instruction provided by the user.

FIGS. 11A and 11B illustrate an example of another operation. Forexample, as illustrated in FIG. 11A, the user touches the device image66 on the device display screen 62 by using an operator and moves theoperator to the device image 64 in the direction indicated by an arrow72, and thereby links the device image 66 to the device image 64.Accordingly, the terminal apparatus 12 (PC (A)) related to the deviceimage 66 and the image forming apparatus 10 (MFP (B)) related to thedevice image 64 are designated as the target devices that cooperate witheach other, and also the linkage order is designated. In this example, alink is made from the device image 66 to the device image 64, that is,from the terminal apparatus 12 (PC (A)) to the image forming apparatus10 (MFP (B)). With reference to the cooperative function managementtable illustrated in FIG. 7, the order of priority of the print functionis “1” and the order of priority of the scan and transfer function is“2”. In this case, as illustrated in FIG. 11B, the information about theprint function is displayed with priority over (for example, above) theinformation about the scan and transfer function on a cooperativefunction display screen 74.

As described above, according to the first exemplary embodiment, deviceimages related to devices are linked to each other and therebycooperative functions that use functions of the devices are specified.In accordance with the order in which the images are linked, that is,the order in which the devices are linked, the display order ofinformation about the cooperative functions is changed. The linkageorder of the devices is also regarded as the order of functions to beused in the individual devices or the order of movement of data betweenthe devices that cooperate with each other. The operation of linking thedevices (the operation of linking the images) is also regarded as theoperation of designating the order of functions or the order of movementof data. Thus, as a result of changing the display order of theinformation about the cooperative functions in accordance with thelinkage order, the information about the cooperative function that isexpected to be used by the user is preferentially displayed. In otherwords, the information about the cooperative function that is morelikely to be used by the user is preferentially displayed. For example,if a link is made from the MFP (B) to the PC (A), it is expected thatthe user will use the cooperative function of “using the function of theMFP (B) first and then transferring data from the MFP (B) to the PC(A)”. On the other hand, if a link is made from the PC (A) to the MFP(B), it is expected that the user will use the cooperative function of“using the function of the PC (A) first and then transferring data fromthe PC (A) to the MFP (B)”. Thus, as a result of changing the displayorder of the information about the cooperative functions in accordancewith the linkage order of the images, the information about thecooperative function that is more likely to be used by the user ispreferentially displayed. Furthermore, the order of functions to be usedand the order of movement of data are designated without a specialoperation other than an operation of linking the device images, and theinformation about the cooperative function that is expected to be usedby the user is displayed.

Hereinafter, modification examples of the first exemplary embodimentwill be described.

First Modification Example

Hereinafter, a first modification example will be described. In thefirst modification example, a function of a device assigned to acooperative function varies according to a position in a device imagerelated to the device. When a specific position in a device image isdesignated by the user, information about a cooperative function thatuses the function corresponding to the specific position ispreferentially displayed. Hereinafter, the first modification examplewill be described in detail.

FIG. 12 illustrates an example of a device function management tableaccording to the first modification example. The data of the devicefunction management table according to the first modification example isstored as the device function information 32 in the server 14. In thedevice function management table according to the first modificationexample, for example, a device ID, information representing the name(for example, the type) of a device, information representing a positionin an image, information representing the function corresponding to theposition (function identification information), and an image ID areassociated with each other. A position in an image is a specificposition (specific portion) in a device image related to a device, forexample, a specific position in the device image schematicallyrepresenting the device or a specific position in the device imagecaptured by a camera. A different function is associated with eachspecific position in the image.

FIGS. 13A and 13B illustrate an example of a screen displayed on the UIunit 46 of the terminal apparatus 16. For example, it is assumed thatthe image forming apparatus 10 (MFP (B)) and the terminal apparatus 12(PC (A)) are identified as the target devices that cooperate with eachother. As illustrated in FIG. 13A, the device display screen 62 isdisplayed on the UI unit 46 of the terminal apparatus 16, and the deviceimages 64 and 66 are displayed on the device display screen 62. Forexample, in the device image 64, a print function is associated with aspecific position (partial image 76) corresponding to a body portion ofthe MFP (B). In the device image 64, a scan function is associated witha specific position (partial image 78) corresponding to a documentcover, a document glass, and an automatic document feeder of the MFP(B). In the device image 64, a stapling function is associated with aspecific position (partial image 80) corresponding to a post-processor.The stapling function is a function of stapling output sheets. In thedevice image 66, a data storage function is associated with a specificposition (partial image 82) corresponding to a body portion of the PC(A). In the device image 66, a screen display function is associatedwith a specific position (partial image 84) corresponding to a displayof the PC (A). The data storage function is a function of storing datareceived from another apparatus in the PC (A). The screen displayfunction is a function of displaying data received from anotherapparatus in the PC (A).

The controller 48 of the terminal apparatus 16 may cause the names offunctions associated with specific positions in a device image (forexample, print, scan, and so forth) to be displayed on the devicedisplay screen 62. Accordingly, the user is provided with informationclearly representing the correspondence between specific positions andfunctions. Of course, the names of functions are not necessarilydisplayed.

In the first modification example, if positions associated withfunctions in device images are designated by the user, the functionsassociated with the designated positions are designated as the targetfunctions that cooperate with each other. The user links, by using anindicator, the specific positions (partial images) associated with thefunctions in the device images representing the target devices thatcooperate with each other. For example, as indicated by an arrow 86, theuser touches, by using an operator, the partial image 78 on the devicedisplay screen 62 and moves the operator to the partial image 84,thereby linking the partial image 78 to the partial image 84.Accordingly, the MFP (B) related to the device image 64 including thepartial image 78 and the PC (A) related to the device image 66 includingthe partial image 84 are designated as the target devices that cooperatewith each other, the scan function corresponding to the partial image 78and the screen display function corresponding to the partial image 84are designated, and the linkage order thereof is designated. The orderin which the partial images are linked corresponds to the linkage order.In the example illustrated in FIG. 13A, a link is made from the partialimage 78 to the partial image 84, that is, from the MFP (B) to the PC(A). The scan function and the screen display function are designated asthe functions used for a cooperative function. The informationrepresenting the linkage order of the devices and the informationrepresenting the specific positions designated in the device images bythe user are transmitted from the terminal apparatus 16 to the server14.

When the target devices that cooperate with each other (for example, thePC (A) and the MFP (B)) are identified, the specifying unit 38 of theserver 14 specifies the cooperative functions to be implemented throughcooperation between the PC (A) and the MFP (B) in the cooperativefunction management table illustrated in FIG. 7. Also, the specifyingunit 38 specifies the functions associated with the specific positionsdesignated in the device images by the user with reference to the devicefunction management table illustrated in FIG. 12. Also, the specifyingunit 38 assigns a higher priority to the cooperative function that usesthe functions associated with the positions designated by the user andassigns a lower priority to the cooperative function that does not usethe functions, among the cooperative functions to be implemented throughcooperation between the PC (A) and the MFP (B).

The information about the cooperative functions specified in theabove-described manner and the information representing the order ofpriority are transmitted from the server 14 to the terminal apparatus16. The controller 48 of the terminal apparatus 16 causes the UI unit 46to display the information about the cooperative functions, as theinformation about candidate cooperative functions, in accordance withthe order of priority.

For example, as illustrated in FIG. 13B, the controller 48 of theterminal apparatus 16 causes the display of the UI unit 46 to display acooperative function display screen 88 and to display the informationabout candidate cooperative functions on the cooperative functiondisplay screen 88. Since the scan function and the screen displayfunction are designated in this order by the user, the information abouta scan, transfer, and display function, which is a cooperative functionexecuted through cooperation between the scan function and the screendisplay function, is displayed with priority over (for example, above)the information about the other cooperative functions. For example, theinformation about the scan, transfer, and display function is displayedwith priority over the information about a scan, transfer, and storagefunction, which is a cooperative function executed through cooperationbetween the scan function and the data storage function. The scan,transfer, and display function is a function of transferring the datagenerated through scanning by the MFP (B) to the PC (A) and displayingthe data on the screen of the PC (A). The scan, transfer, and storagefunction is a function of transferring the data generated throughscanning by the MFP (B) to the PC (A) and storing the data in the PC(A). In the example illustrated in FIG. 13B, explanations of theindividual cooperative functions are displayed as the information aboutthe individual cooperative functions.

According to the first modification example, in a case where each of thetarget devices that cooperate with each other has plural functions,functions are individually designated and the information about acooperative function that uses the designated functions ispreferentially displayed. Accordingly, a cooperative function that isexpected to be used by the user is preferentially displayed.

FIGS. 14A and 14B illustrate another example of the first modificationexample. FIGS. 14A and 14B are diagrams illustrating an example of ascreen displayed on the UI unit 46 of the terminal apparatus 16. Thecontroller 48 of the terminal apparatus 16 superimposes identifiablecontact region images on specific positions (partial images) to whichspecific functions are assigned in a device image. In the exampleillustrated in FIG. 14A, a contact region image 90 is superimposed onthe partial image 76, a contact region image 92 is superimposed on thepartial image 78, and a contact region image 94 is superimposed on thepartial image 80. Also, a contact region image 96 is superimposed on thepartial image 82, and a contact region image 98 is superimposed on thepartial image 84. Each contact region image is, for example, asemitransparent or opaque image, and has a circular, oval, orrectangular shape. With the contact region images being displayed at thepositions to which specific functions are assigned, the positions arehighlighted, and thus the user may easily determine a position to whicha function is assigned. That is, the user may easily determine theposition that will reacts as a function to be used for a cooperativefunction.

The controller 48 of the terminal apparatus 16 may change the displayform (for example, at least one of the color and shape) of a contractregion image corresponding to a function to be used for a cooperativefunction or a contract region image designated by the user so that thecontact region image is distinguished from the other contact regionimages. For example, as illustrated in FIG. 14B, if the partial image 78is linked to the partial image 84 through an operation indicated by anarrow 100, the contact region image 92 corresponding to the partialimage 78 and the contract region image 98 corresponding to the partialimage 84 are displayed in a color different from the color of the othercontact region images.

FIGS. 15A and 15B illustrate still another example of the firstmodification example. FIGS. 15A and 15B are diagrams illustrating anexample of a screen displayed on the UI unit 46 of the terminalapparatus 16. In a case where an identified device has plural functions,plural functions of the same device may be designated to designate acooperative function to be executed through cooperation between theplural functions. For example, as illustrated in FIG. 15A, it is assumedthat the user links images in the directions indicated by arrows 102,104, and 106. In this case, the partial images 78, 84, 82, and 76 arelinked in this order. The specifying unit 38 of the server 14 specifiesthe functions assigned to the partial images 78, 84, 82, and 76 (scanfunction, screen display function, data storage function, and printfunction), specifies a cooperative function that uses these functions inthe linkage order, and gives a high priority to the cooperativefunction. Accordingly, the information about the cooperative function isdisplayed with priority over the information about the other cooperativefunctions. FIG. 15B illustrates a cooperative function display screen108, and the information about the cooperative function ispreferentially displayed on the cooperative function display screen 108.The cooperative function given priority is, for example, a function of“transferring the data generated through scanning by the MFP (B) to thePC (A), displaying the data on the screen of the PC (A), adjusting colorin the PC (A), storing the data in the PC (A) and transferring the datato the MFP (B), and printing the data in the MFP (B)”.

Second Modification Example

Hereinafter, a second modification example will be described withreference to FIGS. 16A to 17C. FIGS. 16A to 17C illustrate an example ofa screen displayed on the terminal apparatus 16. In the secondmodification example, if a device image is kept pressed for more than apreset time period, the mode shifts to a plural functions selection mode(the mode for designating plural functions).

Also in the second modification example, the image forming apparatus 10(MFP (B)) and the terminal apparatus 12 (PC (A)) are identified as thetarget devices that cooperate with each other, and the device images 64and 66 are displayed on the device display screen 62 in the terminalapparatus 16 as illustrated in FIG. 16A. For example, it is assumed thatthe user continues designating the device image 64 for more than apreset time period by using an operator. That is, it is assumed that thedevice image 64 is held down. In the example illustrated in FIG. 16A,the portion denoted by reference numeral 110 is held down. In this case,as illustrated in FIG. 16B, a function selection screen 112 is displayedon the UI unit 46 of the terminal apparatus 16, and a list of functionsof the MFP (B) is displayed on the function selection screen 112. In theexample illustrated in FIG. 16B, the names of the individual functions(scan, print, stapling, and FAX transmission) are displayed. Forexample, if the device image 64 is designated by the user, the deviceidentification information of the MFP (B) related to the device image 64is transmitted from the terminal apparatus 16 to the server 14. In theserver 14, the functions of the MFP (B) are specified with reference tothe device function management table. The information about thefunctions (for example, function identification information) istransmitted from the server 14 to the terminal apparatus 16 and isdisplayed on the terminal apparatus 16. Of course, the functions of theMFP (B) may be specified in the terminal apparatus 16. In a case wherethe user interface of the terminal apparatus 16 is able to detect acontact force of an operation performed by the user in plural stages,the function selection screen 112 may be displayed in response todetection of a larger contact force (for example, detection of a presetcontact force).

On the function selection screen 112 illustrated in FIG. 16B, pluralfunctions to be used for a cooperative function are selected by theuser. For example, it is assumed that (1) scan function and (4) FAXtransmission function are selected. In this case, a confirmation screen114 is displayed on the UI unit 46 of the terminal apparatus 16, asillustrated in FIG. 16C. On the confirmation screen 114, the informationrepresenting the functions selected by the user is displayed. If theuser designates the order of cooperation (the order of use) on theconfirmation screen 114, another confirmation screen 116 is displayed onthe UI unit 46 of the terminal apparatus 16, as illustrated in FIG. 17A.In the example illustrated in FIG. 17A, the order of (1) scan functionis “1” and the order of (4) FAX transmission function is “2”. On thefunction selection screen 112 illustrated in FIG. 16B, the pluralfunctions to be used for a cooperative function may be selected by theuser in accordance with the order of cooperation. In this case, displayof the confirmation screen 114 illustrated in FIG. 16C is omitted, andthe confirmation screen 116 illustrated in FIG. 17A is displayed on theUI unit 46 of the terminal apparatus 16.

Subsequently, if the user presses a “YES” button on the confirmationscreen 116 illustrated in FIG. 17A, the functions of the MFP (B) to beused for a cooperative function are confirmed, and the screen shifts tothe device display screen 62, as illustrated in FIG. 17B. On the devicedisplay screen 62, the user links the device image 64 to the deviceimage 66, as indicated by an arrow 118. Accordingly, the PC (A) isdesignated as the device to be used for a cooperative function and theorder of cooperation (the order of use) is designated. The specifyingunit 38 of the server 14 specifies the cooperative function that usesthe individual functions in the order of cooperation and gives a higherpriority to the cooperative function. Accordingly, the information aboutthe cooperative function is displayed with priority over the informationabout the other cooperative functions. The information about thecooperative function is transmitted from the server 14 to the terminalapparatus 16 and is displayed on the terminal apparatus 16. For example,as illustrated in FIG. 17C, a cooperative function display screen 120 isdisplayed on the UI unit 46 of the terminal apparatus 16, and theinformation about the cooperative function is displayed on thecooperative function display screen 120. The information about thecooperative function that uses the individual functions in accordancewith the order of cooperation is preferentially displayed. Thecooperative function given priority is, for example, a function of“transmitting the image generated through scanning by the MFP (B) by FAXand transferring the copy of the transmitted data to PC (A)”.

If the user presses a “NO” button on the confirmation screen 116illustrated in FIG. 17A, the selection of the functions to be used forthe cooperative function of the MFP (B) is cancelled, and the screenshifts to the function selection screen 112. A “cancel” button isdisplayed on the function selection screen 112. If the user presses the“cancel” button, the screen shifts to the device display screen 62.

According to the second modification example, a list of functions of adesignated device is displayed, and thus the user may easily recognizethe functions of the device.

FIG. 18 illustrates another example of the second modification example.In the second modification example, a button image 122 for selectingplural functions is displayed on the device display screen 62. If theuser presses the button image 122 while pressing a device image, themode shifts to the plural functions selection mode. In the exampleillustrated in FIG. 18, the button image 122 is pressed while the deviceimage 64 related to the MFP (B) being pressed by the user, as denoted byreference numeral 124. In this case, the screen shifts to the functionselection screen 112 illustrated in FIG. 16B. After that, the processdescribed above with reference to FIGS. 16A to 17C is performed.

Third Modification Example

Hereinafter, a third modification example will be described withreference to FIGS. 19A and 19B. FIGS. 19A and 19B illustrate an exampleof a screen displayed on the UI unit 46 of the terminal apparatus 16.

Also in the third modification example, the image forming apparatus 10(MFP (B)) and the terminal apparatus 12 (PC (A)) are identified as thetarget devices that cooperate with each other, and the device images 64and 66 are displayed on the device display screen 62 in the terminalapparatus 16 as illustrated in FIG. 19A. Also, the partial images 78,84, 82, and 76 are linked in this order as indicated by arrows 126, 128,and 130. That is, the scan function, the screen display function, thedata storage function, and the print function are designated in thisorder.

In the third modification example, if, in terms of a linkage order, animage immediately before the target image to be cancelled is designated,the function related to the target image to be cancelled is eliminatedfrom a cooperative function. That is, the target function to becancelled is eliminated from the target functions that cooperate witheach other. For example, if, in terms of a linkage order, an imageimmediately before the target image to be cancelled is designated, theinformation representing the designation is transmitted from theterminal apparatus 16 to the server 14, and the specifying unit 38 ofthe server 14 eliminates the function related to the target image to becancelled from the target functions that cooperate with each other.

With reference to the example illustrated in FIG. 19A, a link is madefrom the partial image 76 to the partial image 82 as indicated by abroken-line arrow 132. That is, the partial image 82 immediately beforethe partial image 76 in the linkage order is designated. In this case,the function related to the partial image 76 (print function) iseliminated from the target functions that cooperate with each other.

After that, as illustrated in FIG. 19B, if a link is made from thepartial image 82 to the partial image 80 by the user in the directionindicated by an arrow 134, the function related to the partial image 80(stapling function) is selected as the target function that cooperateswith another function, and the cooperative function that uses thefunction related to the partial image 80 is preferentially selected.

According to the third modification example, the target function to becancelled is eliminated from the target functions that cooperate witheach other, and the cooperative function that uses the target functionto be cancelled is cancelled, without a special operation other than anoperation of linking images. For example, if a target function thatcooperates with another function is designated by mistake, the functiondesignated by mistake is cancelled by designating the image immediatelybefore the image related to the function in the linkage order, withoutperforming other special operations. Accordingly, the target function tobe cancelled is eliminated with a simple operation compared to the caseof eliminating the function by using a cancel button or the like.

A trail of a cancellation operation (for example, the trail indicated bythe arrow 132) may be displayed on the device display screen 62. In thiscase, the trail of the cancellation operation may be displayed in a formdifferent from the trails of linkage operations (for example, the trailsindicated by the arrows 126, 128, and 130), for example, in a differentcolor or with a different type of line.

Alternatively, in the case of cancelling cooperation, the user maydesignate the device image or partial image related to the target deviceto be cancelled or may press a cooperation cancellation button on thedevice display screen 62. In a case where a device that is not thetarget of cooperation is displayed on the device display screen 62, theuser may eliminate the device from the target devices that cooperatewith each other by designating the device image related to the device onthe device display screen 62. Alternatively, the device may beeliminated from the target devices that cooperate with each other if apreset operation is performed, for example, if a cross mark or the likeis attached to the device image related to the target device to becancelled.

Fourth Modification Example

Hereinafter, a fourth modification example will be described withreference to FIGS. 20A and 20B. FIGS. 20A and 20B illustrate an exampleof a screen displayed on the UI unit 46 of the terminal apparatus 16.

Also in the fourth modification example, the image forming apparatus 10(MFP (B)) and the terminal apparatus 12 (PC (A)) are identified as thetarget devices that cooperate with each other, and the device images 64and 66 are displayed on the device display screen 62 in the terminalapparatus 16 as illustrated in FIG. 20A. Also, the partial images 78 and84 (contact region images 92 and 98) are linked in this order asindicated by an arrow 136. That is, the scan function and the screendisplay function are designated in this order.

In the fourth modification example, the controller 48 of the terminalapparatus 16 causes the field of view of the camera 42 to be changed andcauses the changed field of view to be displayed on the device displayscreen 62 while causing a device image representing the devicedesignated last as the target device that cooperates to be displayed onthe device display screen 62. In this case, the pieces of identificationinformation of the devices that have been designated as the targetdevices that cooperate with each other (for example, the MFP (B) and thePC (A)) are stored in the memory of the terminal apparatus 16 or theserver 14.

In the example illustrated in FIG. 20A, the terminal apparatus 12 (PC(A)) is the device that is designated last as the target device thatcooperates, and thus the device image 66 thereof remains on the devicedisplay screen 62. If the user changes the field of view of the camera42 of the terminal apparatus 16, the controller 48 of the terminalapparatus 16 causes an image captured by the camera 42 to be displayedon the device display screen 62 while causing the device image 66 to bedisplayed on the device display screen 62. For example, in the case ofusing a projector (C), which is not within the field of view in thestate illustrated in FIG. 20A, as the target device that cooperates, theuser changes the orientation of the camera 42 so that the projector (C)is included in the field of view of the camera 42, with the device image66 being displayed on the device display screen 62 as illustrated inFIG. 20B, and then captures an image of the projector (C). Accordingly,the image of the projector (C) is captured by the camera 42, and adevice image 138 related to the projector (C) is displayed on the devicedisplay screen 62. The projector (C) is identified and, for example, thename of a function of the projector (C) (for example, “projection”) isdisplayed on the device display screen 62. Also, a contact region image140 is superimposed on the device image 138 representing the projector(C). As indicated by an arrow 142, if a link is made from the partialimage 84 (contact region image 98) to the device image 138 (contactregion image 140) by the user, the function of the device related to thedevice image 138 (for example, a projection function) is selected as thetarget function that cooperates, and a cooperative function that usesthe function related to the device image 138 is preferentiallydisplayed.

According to the fourth modification example, even if an image of pluraltarget devices that cooperate with each other is not captured at onetime, individual images of the plural devices may be captured and theplural devices may cooperate with each other. That is, the targetdevices that cooperate with each other are not necessarily located closeto each other. According to the fourth modification example, the imagesof the plural devices may be separately captured and the plural devicesmay cooperate with each other even in such a case. Of course, the angleof view of the camera 42 may be changed or the field of view may beincreased or decreased. If these operations are insufficient, imagecapturing may be performed plural times to identify the individualdevices.

In the above-described example, a device image related to the devicedesignated last as the target device that cooperates remains on thedevice display screen 62. Alternatively, a device image designated bythe user, a device image related to the device whose image is capturedfirst, or a device image related to a preset basic device may remain onthe device display screen 62. Also in this case, images of pluraldevices may be separately captured and the devices may cooperate witheach other.

In the first exemplary embodiment and the first to fourth modificationexamples described above, a device image representing an identifieddevice, a contact region image, a trail of a linkage operation (forexample, a trail of movement), and a trail of a cancellation operationmay be displayed three-dimensionally so that they are distinguished froma background image. That is, these images may be displayed asthree-dimensional images. For example, a background image may betwo-dimensionally displayed, whereas a device image, a contact regionimage, and a trail of an operation may be three-dimensionally displayed.Accordingly, the visibility of the device image, the contact regionimage, and the trail of an operation may increase. Alternatively, adevice image related to an unidentified device and a background imageare not displayed on the device display screen 62, and only a deviceimage related to an identified device may be displayed on the devicedisplay screen 62.

As described above, when images related to target devices are linked toeach other and a cooperative function is specified, a connection requestis transmitted from the terminal apparatus 16 to the target devices thatcooperate with each other, and the terminal apparatus 16 is connected tothe target devices. Hereinafter, the connection process will bedescribed with reference to FIG. 21. FIG. 21 is a sequence diagramillustrating the process.

First, the user provides an instruction to start an application(program) for executing a cooperative function by using the terminalapparatus 16. In response to the instruction, the controller 48 of theterminal apparatus 16 starts the application (S01). The application maybe stored in the memory 44 of the terminal apparatus 16 in advance ormay be downloaded from the server 14 or the like.

Subsequently, the target devices that cooperate with each other areidentified by using the marker-based AR technology, the markerless ARtechnology, or the position information AR technology (S02). Of course,the target devices may be identified by using a technology other thanthe AR technologies. In the case of using the marker-based AR technologyor the markerless AR technology, the user captures an image of thetarget devices by using the camera 42 of the terminal apparatus 16. Forexample, in the case of using the image forming apparatus 10 (MFP (B))and the terminal apparatus 12 (PC (A)) as the target devices, the usercaptures an image of the image forming apparatus 10 and the terminalapparatus 12 by using the camera 42. Accordingly, the pieces of deviceidentification information of the image forming apparatus 10 and theterminal apparatus 12 are obtained, and the image forming apparatus 10and the terminal apparatus 12 are identified as the target devices. Inthe case of using the position information AR technology, the pieces ofposition information of the image forming apparatus 10 and the terminalapparatus 12 are obtained, the pieces of device identificationinformation of the image forming apparatus 10 and the terminal apparatus12 are specified on the basis of the pieces of position information, andthe image forming apparatus 10 and the terminal apparatus 12 areidentified.

By using the terminal apparatus 16, the user links the plural deviceimages related to the plural target devices that cooperate with eachother, and thereby cooperative functions are specified and the order ofpriority of the cooperative functions is specified. The informationabout the cooperative functions is displayed on the UI unit 46 of theterminal apparatus 16 in accordance with the order of priority (S03).The process of specifying the cooperative functions and the process ofspecifying the order of priority may be performed by the server 14 orthe terminal apparatus 16.

Subsequently, if the user designates, by using the terminal apparatus16, the target cooperative function to be executed, the terminalapparatus 16 transmits information representing a connection request tothe target devices (for example, the image forming apparatus 10 and theterminal apparatus 12) that execute the cooperative function (S04). Forexample, if the pieces of address information representing the addressesof the target devices that cooperate with each other are stored in theserver 14, the terminal apparatus 16 obtains the pieces of addressinformation from the server 14. If the pieces of address information areincluded in the pieces of device identification information, theterminal apparatus 16 may obtain the pieces of address information fromthe pieces of device identification information of the target devices.Alternatively, the pieces of address information of the target devicesmay be stored in the terminal apparatus 16. Of course, the terminalapparatus 16 may obtain the pieces of address information of the targetdevices by using another method. By using the pieces of addressinformation of the target devices (for example, the image formingapparatus 10 and the terminal apparatus 12), the terminal apparatus 16transmits information representing a connection request to the targetdevices (for example, the image forming apparatus 10 and the terminalapparatus 12).

The image forming apparatus 10 and the terminal apparatus 12 that havereceived the information representing a connection request permit or donot permit the connection to the terminal apparatus 16 (S05). Forexample, if the image forming apparatus 10 and the terminal apparatus 12are devices that are not permitted to make a connection or if the numberof devices to which a connection is requested exceeds an upper limit,the connection is not permitted. If the connection from the terminalapparatus 16 is permitted, an operation of changing setting informationunique to the image forming apparatus 10 and the terminal apparatus 12may be prohibited so that the setting information is not changed. Forexample, change of color parameters or setting time to shift to a powersaving mode of the image forming apparatus 10 may be prohibited.Accordingly, the security for the target devices that cooperate witheach other increases. Alternatively, in the case of causing devices tocooperate with each other, change of setting information may be limitedcompared to the case of using each device alone without cooperation withanother device. For example, change of fewer setting items may bepermitted than in the case of using the device alone. Alternatively,viewing of personal information of other users, such as an operationhistory, may be prohibited. Accordingly, the security for personalinformation of users may increase.

Result information representing permission or non-permission ofconnection is transmitted from the image forming apparatus 10 and theterminal apparatus 12 to the terminal apparatus 16 (S06). If theconnection to the image forming apparatus 10 and the terminal apparatus12 is permitted, communication is established between the terminalapparatus 16 and each of the image forming apparatus 10 and the terminalapparatus 12.

Subsequently, the user provides an instruction to execute thecooperative function by using the terminal apparatus 16 (S07). Inresponse to the instruction, execution instruction informationrepresenting the instruction to execute the cooperative function istransmitted from the terminal apparatus 16 to the image formingapparatus 10 and the terminal apparatus 12 (S08). The executioninstruction information transmitted to the image forming apparatus 10includes information representing the process to be executed in theimage forming apparatus 10 (for example, job information), and theexecution instruction information transmitted to the terminal apparatus12 includes information representing the process to be executed in theterminal apparatus 12 (for example, job information).

In response to the execution instruction information, the image formingapparatus 10 and the terminal apparatus 12 execute the individualfunctions (S09). For example, if the cooperative function includes aprocess of transmitting/receiving data between the image formingapparatus 10 and the terminal apparatus 12, as in the scan and transferfunction of transferring scan data from the image forming apparatus 10(MFP (B)) to the terminal apparatus 12 (PC (A)), communication isestablished between the image forming apparatus 10 and the terminalapparatus 12. In this case, for example, the execution instructioninformation transmitted to the image forming apparatus 10 includes theaddress information of the terminal apparatus 12, and the executioninstruction information transmitted to the terminal apparatus 12includes the address information of the image forming apparatus 10. Thecommunication is established between the image forming apparatus 10 andthe terminal apparatus 12 by using these pieces of address information.

After the execution of the cooperative function is finished, theinformation indicating that the execution of the cooperative function iscompleted is transmitted from the image forming apparatus 10 and theterminal apparatus 12 to the terminal apparatus 16 (S10). Theinformation indicating that the execution of the cooperative function iscompleted is displayed on the UI unit 46 of the terminal apparatus 16(S11). If the information indicating that the execution of thecooperative function is completed is not displayed even when a presettime period elapses from the time point at which the executioninstruction is provided, the controller 48 of the terminal apparatus 16may cause the UI unit 46 to display information representing an error,and may transmit execution instruction information or informationrepresenting a connection request to the image forming apparatus 10 andthe terminal apparatus 12 again.

Subsequently, the user determines whether or not to cancel thecooperation state of the image forming apparatus 10 and the terminalapparatus 12 (S12), and a process is performed in accordance with thedetermination result (S13). In the case of cancelling the cooperationstate, the user provides a cancellation instruction by using theterminal apparatus 16. Accordingly, the communication between theterminal apparatus 16 and each of the image forming apparatus 10 and theterminal apparatus 12 is stopped. Also, the communication between theimage forming apparatus 10 and the terminal apparatus 12 is stopped. Inthe case of not cancelling the cooperation state, an executioninstruction may be continuously provided.

Furthermore, the number of target devices that cooperate with each othermay be increased. For example, the device identification information ofthe third device may be obtained, and a cooperative function to beexecuted through cooperation among the three devices including the imageforming apparatus 10 and the terminal apparatus 12 may be specified. Theinformation indicating that the image forming apparatus 10 and theterminal apparatus 12 have already been identified as the target devicesthat cooperate with each other is stored in the server 14 or theterminal apparatus 16.

The pieces of device identification information representing the targetdevices that cooperate with each other and the cooperative functionidentification information representing an executed cooperative functionmay be stored in the terminal apparatus 16 or the server 14. Forexample, the user account information (user identification information)of the user who uses the terminal apparatus 16 may be obtained, andhistory information representing the correspondence among the useraccount information, the pieces of device identification informationrepresenting the target devices that cooperate with each other, and thecooperative function identification information representing theexecuted cooperative function may be created and stored in the terminalapparatus 16 or the server 14. The history information may be created bythe terminal apparatus 16 or the server 14. With reference to thehistory information, the cooperative function that has been executed andthe devices used for the cooperative function are specified.

The target devices that cooperate with each other (for example, theimage forming apparatus 10 and the terminal apparatus 12) may store, ashistory information, the user account information of the user who hasrequested connection and the terminal identification informationrepresenting the terminal apparatus 16 that has requested connection.With reference to the history information, the user who has used thedevices is specified. The user may be specified by using the historyinformation in the case of, for example, specifying the user who wasusing a device when the device was broken, or performing a chargingprocess for consumables or the like. The history information may bestored in the server 14 or the terminal apparatus 16 or may be stored inanother apparatus.

The user account information is stored, for example, in the memory 44 ofthe terminal apparatus 16 in advance. The controller 48 of the terminalapparatus 16 functions as an example of a user identifying unit, readsthe user account information of the user from the memory 44, andidentifies the user who uses the terminal apparatus 16. If the pieces ofuser account information of plural users are stored in the memory 44,the user designates his/her user account information by using theterminal apparatus 16. Accordingly, the user account information of theuser is read and the user is identified. Alternatively, the controller48 of the terminal apparatus 16 may identify the user by reading theuser account information of the user who is logged in to the terminalapparatus 16. Alternatively, if only one piece of user accountinformation is stored in the same terminal apparatus 16, the controller48 of the terminal apparatus 16 may identify the user by reading the onepiece of user account information. If a user account is not set and ifuser account information is not created, initial setting is performedand thereby user account information is created.

Usage histories of cooperative functions may be managed for individualusers, and the information representing the cooperative functionspreviously used by the user represented by read user account informationmay be displayed on the UI unit 46 of the terminal apparatus 16. Theinformation representing the usage history may be stored in the terminalapparatus 16 or the server 14. Also, the information representing acooperative function that is used at a preset frequency or more may bedisplayed. With such a shortcut function being provided, a useroperation regarding a cooperative function may be reduced.

Second Exemplary Embodiment

Hereinafter, an image forming system serving as an informationprocessing system according to a second exemplary embodiment of thepresent invention will be described. FIG. 22 illustrates a server 14Aaccording to the second exemplary embodiment. The image forming systemaccording to the second exemplary embodiment includes the server 14Ainstead of the server 14 according to the first exemplary embodiment.Except for the server 14A, the configuration of the image forming systemaccording to the second exemplary embodiment is the same as that of theimage forming system according to the first exemplary embodimentillustrated in FIG. 1.

The server 14A manages cooperative functions, each being executedthrough cooperation between plural functions, like the server 14according to the first exemplary embodiment. Hereinafter, theconfiguration of the server 14A will be described in detail withreference to FIG. 22.

The server 14A includes a communication unit 28, a memory 30A, and acontroller 36A. The communication unit 28 has the same configuration asthat of the communication unit 28 according to the first exemplaryembodiment.

The memory 30A is a storage apparatus such as a hard disk or an SSD. Thememory 30A stores, for example, device function information 32,cooperative function information 144, various pieces of data, variousprograms, pieces of device address information representing theaddresses of individual devices, and so forth. Of course, these piecesof information and data may be stored in different storage apparatusesor in one storage apparatus. The device function information 32 and thecooperative function information 144 stored in the memory 30A may beprovided to the terminal apparatus 16 periodically or at designatedtiming, and accordingly the information stored in the terminal apparatus16 may be updated.

The device function information 32 is the same information as the devicefunction information 32 according to the first exemplary embodiment.

The cooperative function information 144 is information representing acooperative function to be executed through cooperation between pluralfunctions, and is, for example, information representing thecorrespondence between information representing a combination of piecesof function identification information for identifying functions thatcooperate with each other to execute the cooperative function andcooperative function identification information for identifying thecooperative function. In a case where plural devices cooperate with eachother, the cooperative function information 144 is informationrepresenting a cooperative function to be executed through cooperationbetween plural functions of the plural devices, and is informationrepresenting the correspondence between information representing acombination of pieces of device identification information foridentifying the devices that cooperate with each other to execute thecooperative function and cooperative function identificationinformation.

The controller 36A controls the operations of the individual units ofthe server 14A. The controller 36A includes a specifying unit 38 and acooperative function registration unit 146.

The specifying unit 38 identifies a device on the basis of deviceidentification information and specifies a function of the device, likethe specifying unit 38 according to the first exemplary embodiment.Also, the specifying unit 38 specifies a cooperative function to beexecuted through cooperation between functions of target devices thatcooperate with each other. Also in the second exemplary embodiment,device identification information is obtained and a device is identifiedby using the AR technologies. Of course, device identificationinformation may be obtained and a device may be identified by using atechnology other than the AR technologies.

In the second exemplary embodiment, device images related to the targetdevices that cooperate with each other are displayed on the UI unit 46of the terminal apparatus 16, a cooperative function is specified bysuperimposing the device images on one another, and the informationabout the cooperative function is displayed on the UI unit 46 of theterminal apparatus 16.

The cooperative function registration unit 146 has a function ofregistering a cooperative function. For example, the cooperativefunction registration unit 146 associates registration information of atarget cooperative function to be registered and an image to bedisplayed for executing the cooperative function (for example, ashortcut image) with each other. The image is, for example, an imagegenerated by superimposing device images on one another or an imageschematically representing the target devices of superimposing.

Hereinafter, the image forming system according to the second exemplaryembodiment will be described in detail.

First, the cooperative function information 144 will be described indetail with reference to FIG. 23. FIG. 23 illustrates an example of acooperative function management table, which is the cooperative functioninformation 144. In the cooperative function management table, forexample, information representing a combination of device IDs,information representing the names (for example, the types) of thetarget devices that cooperate with each other, and informationrepresenting a cooperative function (cooperative function identificationinformation) are associated with each other. Unlike the cooperativefunction management table according to the first exemplary embodimentillustrated in FIG. 7, the cooperative function management tableaccording to the second exemplary embodiment does not includeinformation representing a linkage order and information representing anorder of priority.

Hereinafter, an operation for specifying a cooperative function will bedescribed in detail with reference to FIGS. 24A to 25B. FIGS. 24A to 25Billustrate an example of a screen displayed on the UI unit 46 of theterminal apparatus 16.

For example, as in the first exemplary embodiment, it is assumed thatthe image forming apparatus 10 (MFP (B)) and the terminal apparatus 12(PC (A)) are identified as the target devices that cooperate with eachother by using the AR technologies or the like. As illustrated in FIG.24A, the device display screen 62 is displayed on the UI unit 46 of theterminal apparatus 16, and the device images 64 and 66 related to theidentified devices (the image forming apparatus 10 and the terminalapparatus 12) are displayed on the device display screen 62. In thisstate, the user superimposes the device image related to one of thetarget devices on the device image related to the other target device byusing an indicator (for example, a user's finger, a pen, or a stylus).For example, as illustrated in FIG. 24B, the user designates the deviceimage 64 by using an operator and superimposes the device image 64 onthe device image 66 as indicated by an arrow 148. For example, the usersuperimposes the device images on one another by performing adrag-and-drop operation. Specifically, the user drags the device image64 and drops it at the position where the device image 64 issuperimposed on the device image 66. The drag-and-drop operation is atechnique according to the related art. Alternatively, the device imagesto be superimposed on one another (the images related to the targetdevices that cooperate with each other) may be designated in accordancewith a voice instruction provided by the user. For example, the deviceimages 64 and 66 may be designated as the target device images and maybe superimposed on one another in accordance with a voice instructionprovided by the user.

As a result of superimposing the device images 64 and 66 on one another,the image forming apparatus 10 (MFP (B)) related to the device image 64and the terminal apparatus 12 (PC (A)) related to the device image 66are designated as the target devices that cooperate with each other. Inthe example illustrated in FIG. 24B, the device image 64 is superimposedon the device image 66. Alternatively, the device image 66 may besuperimposed on the device image 64.

The controller 48 of the terminal apparatus 16 may cause a device imagethat is being dragged to be displayed on the UI unit 46 in anidentifiable manner. For example, a device image that is being draggedmay be displayed translucently or in a specific color.

When the device image 64 is superimposed on the device image 66, aconfirmation screen 150 is displayed on the UI unit 46 of the terminalapparatus 16 as illustrated in FIG. 24C. The confirmation screen 150 isa screen for confirming whether or not to cause the designated devicesto cooperate with each other. If a cooperation instruction is providedby the user on the confirmation screen 150 (for example, if a “YES”button is pressed by the user), information about cooperative functionsis displayed on the UI unit 46 of the terminal apparatus 16.Specifically, if the target devices that cooperate with each other (forexample, the image forming apparatus 10 and the terminal apparatus 12)are designated and a cooperation instruction is provided, the specifyingunit 38 of the server 14A specifies the cooperative functions associatedwith the combination of the PC (A) and the MFP (B) in the cooperativefunction management table (cooperative function information 144)illustrated in FIG. 23. Accordingly, the cooperative functions to beexecuted through cooperation between the PC (A) and the MFP (B) arespecified. The information about the cooperative functions specified inthis manner is transmitted from the server 14 to the terminal apparatus16 and is displayed as the information about candidate cooperativefunctions on the UI unit 46 of the terminal apparatus 16.

For example, as illustrated in FIG. 25A, the controller 48 of theterminal apparatus 16 causes the UI unit 46 to display a cooperativefunction display screen 152 and to display the information about thecandidate cooperative functions on the cooperative function displayscreen 152. With the PC (A) and the MFP (B) being caused to cooperatewith each other, for example, a scan and transfer function and a printfunction are implemented. Thus, the information about the scan andtransfer function and the information about the print function aredisplayed on the cooperative function display screen 152.

If a cooperative function is designated by the user and an executioninstruction is provided by the user, a connection request is transmittedfrom the terminal apparatus 16 to the target devices that cooperate witheach other. As illustrated in FIG. 25B, a waiting screen 154 isdisplayed on the UI unit 46 of the terminal apparatus 16 while theconnection request is being processed. When the connection between theterminal apparatus 16 and the target devices is successfullyestablished, the designated cooperative function is executed.

As described above, according to the second exemplary embodiment, deviceimages related to devices are superimposed on one another, and therebycooperative functions that use the functions of the devices arespecified. Thus, the functions may be caused to cooperate with eachother without a special operation other than an image operation, and thefunctions may be caused to cooperate with each other with a simpleoperation.

Hereinafter, modification examples of the second exemplary embodimentwill be described.

Fifth Modification Example

Hereinafter, a fifth modification example will be described withreference to FIGS. 26A to 27. FIGS. 26A to 27 illustrate an example of ascreen displayed on the UI unit 46 of the terminal apparatus 16. In thefifth modification example, plural device images are superimposed on oneanother and then a new image representing the superimposed state isgenerated and displayed. Hereinafter, the fifth modification examplewill be described in detail.

Also in the fifth modification example, it is assumed that the imageforming apparatus 10 (MFP (B)) and the terminal apparatus 12 (PC (A))are identified as the target devices that cooperate with each other, andthat the device images 64 and 66 are displayed on the device displayscreen 62 in the terminal apparatus 16 as illustrated in FIG. 26A. Forexample, it is assumed that the user superimposes the device image 64 onthe device image 66 by using an operator as indicated by an arrow 156.As a result of superimposing the device images 64 and 66 on one another,the image forming apparatus 10 (MFP (B)) related to the device image 64and the terminal apparatus 12 (PC (A)) related to the device image 66are designated as the target devices that cooperate with each other.

When the device image 64 is superimposed on the device image 66, aconfirmation screen 158 is displayed on the UI unit 46 of the terminalapparatus 16 as illustrated in FIG. 26B. The confirmation screen 158 isa screen for confirming whether or not to cause the designated devicesto cooperate with each other and is a screen for confirming whether ornot there is another device that is to be caused to cooperate. If aninstruction to cause the MFP (B) to cooperate with the PC (A) isprovided by the user (for example, if a “YES” button to cause the MFP(B) to cooperate with the PC (A) is pressed by the user), the MFP (B) iscaused to cooperate with the PC (A). If there is another device that isto be caused to cooperate (for example, if a “YES” button to causeanother device to cooperate is pressed by the user), the device displayscreen 62 is displayed as illustrated in FIG. 26C. On the device displayscreen 62, a new device image 160, which is generated by grouping thesuperimposed device images 64 and 66, is displayed. The new device image160 may be generated by, for example, the controller 48 of the terminalapparatus 16 or the controller 36A of the server 14A. The device image160 is an image representing the state in which the device image 64 issuperimposed on the device image 66.

In the fifth modification example, the controller 48 of the terminalapparatus 16 causes the newly generated device image 160 to remain onthe device display screen 62. If the user changes the field of the viewof the camera 42 of the terminal apparatus 16, the controller 48 of theterminal apparatus 16 causes an image captured by the camera 42 to bedisplayed on the device display screen 62, with the device image 160being displayed on the device display screen 62. In this case, thepieces of identification information of the devices that have alreadybeen designated as the target devices that cooperate with each other(for example, the MFP (B) and the PC (A)) are stored in the memory ofthe terminal apparatus 16 or the server 14A.

For example, in the case of using the projector (C), which is not withinthe field of view in the state illustrated in FIG. 26A, as a targetdevice that cooperates, the user changes the orientation of the camera42 so that the projector (C) is included in the field of view of thecamera 42 with the device image 160 being displayed on the devicedisplay screen 62 as illustrated in FIG. 26C, and then captures an imageof the projector (C). Accordingly, the image of the projector (C) iscaptured by the camera 42, the device image 138 related to the projector(C) is displayed on the device display screen 62, and the projector (C)is identified. As indicated by an arrow 162, if the user superimposesthe new device image 160 on the device image 138, the device related tothe device image 138 is selected as the target device that cooperates.If there is no more device that is to be caused to cooperate, acooperative function to be executed through cooperation between thetarget devices that have already been designated is specified by theserver 14, and the information about the cooperative function istransmitted from the server 14 to the terminal apparatus 16 and isdisplayed on the UI unit 46 of the terminal apparatus 16. For example,if a “NO” button to provide an instruction indicating that there is nodevice that is to be caused to cooperate is pressed by the user on theconfirmation screen 158 illustrated in FIG. 26B, the information aboutthe cooperative function is displayed on the UI unit 46 of the terminalapparatus 16.

As described above, according to the fifth modification example, a newdevice image generated by superimposing device images on one another issuperimposed on a device image related to another candidate device thatcooperates, and accordingly the number of devices that cooperate witheach other is increased. The new device image represents a state inwhich the device images related to the target devices that cooperatewith each other are superimposed on one another. Thus, as a result ofdisplaying the new device image, the devices that have been designatedas the target devices that cooperate with each other may be easilyidentified from a visual point of view.

The display of the confirmation screen 158 may be omitted. In this case,if an instruction to display a cooperative function is provided by theuser after the device images are superimposed on one another, theinformation about the cooperative function to be executed by the targetdevices that have been designated through the superimposing and thatcooperate with each other is displayed on the UI unit 46 of the terminalapparatus 16.

In the fifth modification example, a new device image is generated bygrouping device images that are superimposed on one another.Alternatively, the device images that are superimposed on one anothermay be replaced with another image and the image may be displayed on thedevice display screen 62. For example, as illustrated in FIG. 27, a newdevice image 164 is displayed on the device display screen 62 instead ofthe device image 160 generated through grouping, and the new deviceimage 164 is operated as indicated by an arrow 166. The device image 164is an image schematically representing the MFP (B) and the PC (A), animage representing a function of the MFP (B) and a function of the PC(A), or the like. The device image 164 may be generated by thecontroller 48 of the terminal apparatus 16 or the controller 36A of theserver 14A. As a result of displaying an image representing functions asa new device image, the functions designated as the target functionsthat cooperate with each other may be easily identified from a visualpoint of view.

In the example illustrated in FIG. 27, the device image 160 isautomatically replaced with the new device image 164. At the time of thereplacement, a confirmation screen may be displayed on the UI unit 46 ofthe terminal apparatus 16. FIG. 28 illustrates an example of theconfirmation screen. After plural device images are superimposed on oneanother, a confirmation screen 168 is displayed on the UI unit 46 of theterminal apparatus 16. On the confirmation screen 168, the device image160 generated by grouping the plural device images and the schematicdevice image 164 are displayed. If an instruction to replace the imageis provided by the user (for example, if a “YES” button is pressed bythe user), the device image 160 is replaced by the schematic deviceimage 164, and the schematic device image 164 is displayed on the devicedisplay screen 62 as illustrated in FIG. 27.

Sixth Modification Example

Hereinafter, a sixth modification example will be described withreference to FIGS. 29A to 31C. FIGS. 29A to 31C illustrate an example ofa screen displayed on the UI unit 46 of the terminal apparatus 16. Inthe sixth modification example, a device image generated at the lastcooperation is used as an image for a shortcut button when a cooperativefunction is executed next time. Hereinafter, the sixth modificationexample will be described in detail.

In the sixth modification example, as in the fifth modification example,it is assumed that the device images 64 and 66 are superimposed on oneanother, that the image forming apparatus 10 (MFP (B)) and the terminalapparatus 12 (PC (A)) are identified as the target devices thatcooperate with each other, and that the new device image 160 isgenerated by grouping the device images 64 and 66.

If the user provides an instruction to register the new device image 160as an image for a shortcut button by using the terminal apparatus 16, acooperative function registration screen 170 is displayed on the UI unit46 of the terminal apparatus 16 as illustrated in FIG. 29A. On thecooperative function registration screen 170, the new device image 160is displayed, and also registration information of a cooperativefunction to be executed through cooperation between the MFP (B) and thePC (A) is displayed. For example, the name of the cooperative function,the outline of the cooperative function, and the informationrepresenting the devices that are necessary for cooperation aredisplayed as the registration information. The user may edit theregistration information (for example, the name or outline of thecooperative function) by using the terminal apparatus 16.

If an instruction not to register the shortcut function for thecooperative function is provided by the user on the cooperative functionregistration screen 170 (for example, if a “NO” button is pressed by theuser), a confirmation screen 172 is displayed on the UI unit 46 of theterminal apparatus 16 as illustrated in FIG. 29B. If an end instructionis provided by the user on the confirmation screen 172 (for example, ifa “YES” button for end of registration is pressed by the user), thecooperative function registration process ends. If a partial editinstruction is provided by the user (for example, if a “YES” button forpartial edit is pressed by the user), the screen shifts to thecooperative function registration screen 170.

On the other hand, if an instruction to register the shortcut functionfor the cooperative function is provided by the user on the cooperativefunction registration screen 170 (for example, if a “YES” button ispressed by the user), a confirmation screen 174 is displayed on the UIunit 46 of the terminal apparatus 16 as illustrated in FIG. 30A. If aninstruction to create a shortcut button is provided by the user on theconfirmation screen 174 (for example, if a “YES” button is pressed bythe user), an image representing a shortcut button for executing thecooperative function (shortcut button image) is generated. The shortcutbutton image may be created by the terminal apparatus 16 or the server14A. For example, as illustrated in FIG. 30B, a shortcut button image178 is displayed on a screen 176 (for example, a desktop screen) of theUI unit 46 of the terminal apparatus 16. The shortcut button image 178includes the device image 160. For example, the cooperative functionregistration unit 146 of the server 14A associates the registrationinformation of the target cooperative function with the shortcut buttonimage 178. Of course, the association may be performed by the terminalapparatus 16. If plural shortcut button images are generated andregistered, a list of the shortcut button images is displayed on thescreen 176. The registration information of the cooperative function andthe shortcut button image are stored in, for example, the terminalapparatus 16. Alternatively, the registration information of thecooperative function and the shortcut button image may be stored in theserver 14A in association with user account information. In this case,if the user account information is transmitted from the terminalapparatus 16 to the server 14A, the information associated with the useraccount information (the registration information of the cooperativefunction and the shortcut button image) is transmitted from the server14A to the terminal apparatus 16 and is displayed on the terminalapparatus 16.

The shortcut button image 178 includes status images 180 and 182. Eachstatus image is an image representing the status of the target devicethat cooperates, that is, an image representing whether or not thetarget image is in the state of being able to execute the cooperativefunction. For example, the status image 180 is an image representing thestatus of the image forming apparatus 10 (MFP (B)), and the status image182 is an image representing the status of the terminal apparatus 12 (PC(A)). If the target device that cooperates is in the state of being ableto execute the cooperative function, the status image is displayed in aspecific color representing the state (for example, green or blue). Ifthe target device that cooperates is in the state of being unable toexecute the cooperative function, the status image is displayed in aspecific color representing the state (for example, white or red). Inthe example illustrated in FIG. 30B, the status image 180 is displayedin green and the status image 182 is displayed in white. Thus, the imageforming apparatus 10 (MFP (B)) is in the state of being able to executethe cooperative function whereas the terminal apparatus 12 (PC (A)) isin the state of being unable to execute the cooperative function. Ofcourse, the statuses of the target devices that cooperate with eachother may be represented by information other than colors. For example,a status image having a shape corresponding to a status may bedisplayed, or a character string representing a status may be displayed.

The state of being able to execute the cooperative function is, forexample, a state where the device is not being used by another user orthe device is not broken. The state of being unable to execute thecooperative function is, for example, a state where the device is beingused by another user or the device is broken. For example, the terminalapparatus 16 obtains information representing the status of the targetdevice from the target device and displays the status image in aspecific color on the basis of the information. Alternatively, theserver 14A may obtain information representing the status of the targetdevice from the target device, and the terminal apparatus 16 may obtainthe information from the server 14A.

As described above, the cooperative function and the shortcut buttonimage are managed in association with each other. Accordingly, anoperation for executing the cooperative function next time may bereduced.

FIGS. 31A to 31C illustrate an example of a view and edit screen forcooperative functions. If the user provides an instruction to view theregistration information about cooperative functions by using theterminal apparatus 16, a view and edit screen 184 is displayed on the UIunit 46 of the terminal apparatus 16 as illustrated in FIG. 31A. On theview and edit screen 184, shortcut images as display images, theregistration information of cooperative functions (for example, thenames of cooperative functions, the outlines of cooperative functions,and the names of devices to be linked), and so forth are displayed. Forexample, if the user designates an item about the cooperative functionto be edited (for example, a display image, the name of the cooperativefunction, the outline of the cooperative function, or the names ofdevices to be linked) on the view and edit screen 184, a confirmationscreen 186 is displayed on the UI unit 46 of the terminal apparatus 16as illustrated in FIG. 31B. For example, if the name of a cooperativefunction is designated by the user, the confirmation screen 186 forconfirming whether to edit the name of the cooperative function isdisplayed. If a “YES” button is pressed by the user on the confirmationscreen 186, an edit screen 188 is displayed on the UI unit 46 of theterminal apparatus 16 as illustrated in FIG. 31C. On the edit screen188, the name of the cooperative function is edited by the user, andthen the name before edit and the name after edit are displayed on theedit screen 188. If an instruction to reflect the edit is provided bythe user (for example, if a “YES” button is pressed by the user), theedit is reflected. For example, the name after edit is displayed on thedesktop screen of the UI unit 46 of the terminal apparatus 16 togetherwith the shortcut image. The items other than the name of thecooperative function may be edited in a similar manner.

Seventh Modification Example

Hereinafter, a seventh modification example will be described withreference to FIGS. 32A and 32B. FIGS. 32A and 32B illustrate an exampleof a screen displayed on the UI unit 46 of the terminal apparatus 16. Inthe seventh modification example, after the target device imagessuperimposed on one another are selected, a cooperative function iscancelled if the selection is cancelled in a region where the deviceimages are not displayed on the device display screen 62.

For example, it is assumed that, as illustrated in FIG. 32A, the deviceimage 64 is superimposed on the device image 66 and thereby the newdevice image 160 is generated and displayed. In this case, if the userselects the device image 160 and then cancels the selection of thedevice image 160 in a region where no device images are displayed on thedevice display screen 62 as indicated by an arrow 190, the cooperativefunction to be executed through cooperation between the MFP (B) and thePC (A) is cancelled. That is, as a result of dragging the device image160 to a region where no device images are displayed and dropping thedevice image 160 in the region, the cooperative function is cancelledand the state shifts to the state before the devices cooperate with eachother. For example, information representing the drag-and-drop operationis transmitted from the terminal apparatus 16 to the server 14A, and thespecifying unit 38 of the server 14A cancels the cooperative function.Of course, the terminal apparatus 16 may cancel the cooperativefunction.

FIG. 32B illustrates the device display screen 62 after the cooperativefunction is cancelled. In a state where the devices do not cooperatewith each other, the device images 64, 66, and 138 are separatelydisplayed without being superimposed on one another.

As described above, according to the seventh modification example, as aresult of cancelling the selection of device images superimposed on oneanother, the corresponding cooperative function is cancelled withoutperforming another special operation. Thus, the cooperative function maybe easily cancelled compared to a case where a special operation otherthan cancellation of selection is necessary to cancel the cooperativefunction.

Alternatively, in the case of cancelling the cooperation, the user maydesignate device images related to the target devices on the devicedisplay screen 62 or may press a cooperation cancellation button. If adevice other than the target devices that cooperate with each other isdisplayed on the device display screen 62, the user may designate thedevice image related to the device on the device display screen 62 toeliminate the device from the target devices that cooperate with eachother. If a preset operation of attaching a cross mark or the like to adevice image related to the target device of cancellation is performed,the device may be eliminated from the target devices that cooperate witheach other.

Eighth Modification Example

Hereinafter, an eighth modification example will be described withreference to FIGS. 33A and 33B. FIGS. 33A and 33B illustrate an exampleof a screen displayed on the UI unit 46 of the terminal apparatus 16.

Also in the eighth modification example, the image forming apparatus 10(MFP (B)) and the terminal apparatus 12 (PC (A)) are identified as thetarget devices that cooperate with each other, and the device images 64and 66 are displayed on the device display screen 62 in the terminalapparatus 16 as illustrated in FIG. 33A. As indicated by an arrow 192,the device image 64 is superimposed on the device image 66, andaccordingly the new device image 160 is generated and displayed. Theimage forming apparatus 10 (MFP (B)) and the terminal apparatus 12 (PC(A)) are designated as the target devices that cooperate with eachother.

In the eighth modification example, the controller 48 of the terminalapparatus 16 causes the field of view of the camera 42 to be changed andcauses the changed field of view to be displayed on the device displayscreen 62 while causing a device image generated last throughsuperimposing to be displayed on the device display screen 62. In thiscase, the pieces of identification information of the devices thatcooperate with each other (the devices related to the device imagessuperimposed on one another) are stored in the terminal apparatus 16 orthe server 14A.

In the example illustrated in FIG. 33A, the new device image 160generated by superimposing the device image 64 on the device image 66 isdisplayed, and thus the device image 160 remains displayed on the devicedisplay screen 62. For example, in the case of using the projector (C),which is not within the field of view in the state illustrated in FIG.33A, the user changes the orientation of the camera 42 so that theprojector (C) is included in the field of view of the camera 42, withthe device image 160 being displayed on the device display screen 62 asillustrated in FIG. 33B, and then captures an image of the projector(C). Accordingly, the image of the projector (C) is captured by thecamera 42, and the device image 138 related to the projector (C) isdisplayed on the device display screen 62. As a result of superimposingthe device image 160 on the device image 138, the projector (C) isdesignated as the target device that cooperates.

According to the eight modification example, even if an image of pluraltarget devices that cooperate with each other is not captured at onetime, images of the plural devices may be separately captured and theplural devices may cooperate with each other.

In the above-described example, a device image generated last throughsuperimposing remains displayed on the device display screen 62.Alternatively, a device image designated by the user, a device imagerelated to the device whose image is captured first, or a device imagerelated to a preset basic device may remain displayed on the devicedisplay screen 62. Also in this case, images of the plural devices maybe separately captured and the plural devices may cooperate with eachother.

Ninth Modification Example

Hereinafter, a ninth modification example will be described. In theninth modification example, the process according to the first exemplaryembodiment is applied. That is, an order of priority in whichinformation about cooperative functions is displayed is determined inaccordance with an order in which device images are superimposed on oneanother. For example, as illustrated in FIG. 24B, if the device image 64is selected and is superimposed on the device image 66, a link is madefrom the device image 64 to the device image 66. Accordingly, the imageforming apparatus 10 (MFP (B)) related to the device image 64 and theterminal apparatus 12 (PC (A)) related to the device image 66 aredesignated as the target devices that cooperate with each other, andalso the linkage order thereof is designated. The order in which thedevice images are superimposed on one another corresponds to the linkageorder. In the example illustrated in FIG. 24B, a link is made from theimage forming apparatus 10 (MFP (B)) to the terminal apparatus 12 (PC(A)).

In the ninth modification example, the server 14A stores the cooperativefunction information 34 according to the first exemplary embodiment (thecooperative function management table illustrated in FIG. 7). Asdescribed above, when the target devices that cooperate with each other(for example, the image forming apparatus 10 and the terminal apparatus12) are designated, the specifying unit 38 of the server 14A specifiesthe cooperative functions associated with the combination of the PC (A)and the MFP (B) in the cooperative function management table illustratedin FIG. 7 (cooperative function information 34). Accordingly, thecooperative functions to be implemented through cooperation between thePC (A) and the MFP (B) are specified. Furthermore, when the userdesignates the linkage order of the devices, the specifying unit 38specifies the order of priority associated with the linkage order in thecooperative function management table.

The information about the cooperative functions and the informationrepresenting the order of priority specified in the above-describedmanner are transmitted from the server 14A to the terminal apparatus 16.The controller 48 of the terminal apparatus 16 causes the UI unit 46 todisplay the information about the cooperative functions as theinformation about candidate cooperative functions in accordance with theorder of priority.

As described above, according to the ninth modification example,cooperative functions that use functions of devices are specified bysuperimposing device images on one another. Also, the display order ofthe information about the cooperative functions is changed by changingthe order in which the device images are superimposed on one another,that is, the order in which the devices are linked to each other. Thus,the information about the cooperative function that is expected to beused by the user is preferentially displayed without a special operationother than the operation of superimposing device images on one another.

Tenth Modification Example

Hereinafter, a tenth modification example will be described withreference to FIGS. 34A and 34B. FIGS. 34A and 34B illustrate an exampleof a screen displayed on the UI unit 46 of the terminal apparatus 16.

In the tenth modification example, as in the first modification exampleof the first exemplary embodiment, the function of a device assigned toa cooperative function varies according to a position in a device imagerelated to the device. With a specific partial image in a device imagebeing superimposed on a specific partial image in another device image,a cooperative function that uses the functions related to the specificpartial images is specified. Hereinafter, the tenth modification examplewill be described in detail.

Also in the tenth modification example, it is assumed that the imageforming apparatus 10 (MFP (B)) and the terminal apparatus 12 (PC (A))are identified. As illustrated in FIG. 34A, the device display screen 62is displayed on the UI unit 46 of the terminal apparatus 16, and thedevice images 64 and 66 are displayed on the device display screen 62.As in the first modification example of the first exemplary embodiment,a print function is associated with the partial image 76 in the deviceimage 64, a scan function is associated with the partial image 78 in thedevice image 64, and a stapling function is associated with the partialimage 80 in the device image 64. Also, a data storage function isassociated with the partial image 82 in the device image 66, and ascreen display function is associated with the partial image 84 in thedevice image 66. In the tenth modification example, each partial imageis displayed as an image that is able to be moved separately fromanother partial image.

As in the first modification example of the first exemplary embodiment,the names of individual functions (for example, print, scan, and soforth) may be displayed on the device display screen 62, and the contactregion images 90 to 98 may be displayed on the device display screen 62.In this case, the color of the contact region image corresponding to apartial image designated by the user or the color of the contact regionimage corresponding to a superimposed partial image may be changed. Ofcourse, the names of the functions and the contact region images are notnecessarily displayed.

In the tenth modification example, if a partial image is designated bythe user and if the partial image is superimposed on another partialimage, cooperative functions that use the functions related to thesuperimposed partial images are specified, and the information about thecooperative functions is displayed on the UI unit 46 of the terminalapparatus 16. This specification process may be performed by thespecifying unit 38 of the server 14A or the terminal apparatus 16.

For example, as indicated by an arrow 194 in FIG. 34B, if the user dragsthe partial image 78 and drops it on the partial image 84 by using anoperator, the MFP (B) related to the device image 64 including thepartial image 78 and the PC (A) related to the device image 66 includingthe partial image 84 are designated as the target devices that cooperatewith each other, and also the scan function corresponding to the partialimage 78 and the screen display function corresponding to the partialimage 84 are designated as the target functions that cooperate with eachother.

In the server 14A, the functions corresponding to the individual partialimages are managed. For example, identification information foridentifying a partial image, function identification informationrepresenting a function associated with the partial image, andcooperative function identification information representing acooperative function executed through cooperation between functions arestored in the server 14A in association with each other. If a partialimage is selected on the device display screen 62 and is superimposed onanother partial image, the pieces of identification informationrepresenting the partial images superimposed on one another aretransmitted from the terminal apparatus 16 to the server 14A. In theexample illustrated in FIG. 34B, the pieces of identificationinformation representing the partial images 78 and 84 are transmittedfrom the terminal apparatus 16 to the server 14A. The specifying unit 38of the server 14A specifies the functions corresponding to the partialimages 78 and 84 on the basis of the pieces of identificationinformation and specifies a cooperative function that uses thefunctions. The information about the cooperative function is transmittedfrom the server 14A to the terminal apparatus 16 and is displayed on theterminal apparatus 16.

As described above, according to the tenth modification example, in acase where each of the target devices that cooperate with each other hasplural functions, a function is selected in each of the target devices,and information about a cooperative function that uses the designatedfunctions is preferentially displayed. Accordingly, the cooperativefunction expected to be used by the user is preferentially displayed.

Also in the tenth modification example, as in the first exemplaryembodiment, the order of priority in which cooperative functions aredisplayed may be changed in accordance with the order in which partialimages are superimposed on one another. In this case, the informationabout a cooperative function that uses the functions related to thesuperimposed partial images is preferentially displayed.

Eleventh Modification Example

Hereinafter, an eleventh modification example will be described. In theeleventh modification example, as in the tenth modification example, thefunction of a device assigned to a cooperative function varies accordingto a position in a device image related to the device. However, unlikein the tenth modification example, a partial image is not movableseparately from another partial image. In the eleventh modificationexample, as in the example illustrated in FIG. 24B, the entire deviceimage is moved and superimposed on another device image by the user. Atthis time, a cooperative function is changed in accordance with thesuperimposed position. This will be described by using a specificexample with reference to FIG. 34A. For example, it is assumed that theentire device image 64 related to the MFP (B) is operated by the userand is superimposed on the device image 66 related to the PC (A). Inthis case, if the partial image 78 (the partial image related to thescan function) in the device image 64 is superimposed on the partialimage 84 (the partial image related to the screen display function) inthe device image 66, information about a cooperative function that usesthe scan function and the screen display function is preferentiallydisplayed.

Also in the second exemplary embodiment and the fifth to eleventhmodification examples described above, as in the first exemplaryembodiment, the target devices that cooperate with each other areconnected to each other in accordance with the sequence illustrated inFIG. 21.

Furthermore, in the second exemplary embodiment and the fifth toeleventh modification examples described above, device images related toidentified devices and device images superimposed on one another may bethree-dimensionally displayed so as to be distinguished from abackground image. That is, these images may be displayed asthree-dimensional images. For example, the background image istwo-dimensionally displayed and the device images arethree-dimensionally displayed. Accordingly, the visibility of the deviceimages may increase. Furthermore, the color of a device image designatedby the user may be changed or the designated device image may be blinkedso that the designated device image is distinguished from the otherdevice images.

In the first and second exemplary embodiments and the first to eleventhmodification examples described above, a cooperative function isimplemented by identifying hardware units such as devices.Alternatively, functions of software units (application programs) may becaused to cooperate with each other by identifying the software units.For example, cooperative functions may be specified by linking orsuperimposing images such as icons (corresponding to an example ofimages related to functions) regarding functions displayed on a homescreen of a smartphone or a desktop screen of a personal computer, andthe information about the cooperative functions may be displayed. Inthis case, the order of priority in which the cooperative functions aredisplayed may be changed in accordance with the order in which theimages are linked or superimposed.

According to the first and second exemplary embodiments and the first toeleventh modification examples described above, cooperative functionsthat use the functions of the target devices that cooperate with eachother are specified by applying the AR technologies, and informationabout the cooperative functions is displayed. Accordingly, even if theuser does not know which cooperative function may be executed by thetarget devices that cooperate with each other from the appearances ofthe devices, the user is provided with information about the cooperativefunction. Also, a function that is not executable by a single devicebecomes available by causing devices to cooperate with each other, whichmay be convenient. Furthermore, a cooperative function becomes availableonly by identifying the target devices that cooperate with each other byapplying the AR technologies. Thus, the cooperative function becomesavailable through a simple operation compared to a case where the usermanually performs settings for executing the cooperative function, andthe effort of the user may be reduced.

In an environment in which plural devices are used by plural users,information about cooperative functions is displayed on the terminalapparatus 16 of each user. For example, even if a user interface such asa touch screen is removed from a device, the terminal apparatus 16functions as a user interface, and information about the cooperativefunctions is displayed on the terminal apparatus 16 of each user. Inanother case, for example, if the user temporarily uses plural deviceson the go, a user interface suitable for the user, that is, a userinterface that displays information about cooperative functions that usethe functions of the target devices that are designated by the user andthat cooperate with each other is displayed.

In an operation of executing a cooperative function in the first andsecond exemplary embodiments and the first to eleventh modificationexamples described above, if devices have already been identified bycapturing images of the devices in advance or by receiving deviceidentification information and images of the devices obtained throughidentification by another user, the device identification informationand the images of the devices may be stored in the memory of the server14 or the terminal apparatus 16, the device identification informationand the images of the devices may be read from the memory by activatingan edit screen for executing the cooperative function, the images of thedevices may be linked to each other or superimposed on one another, andthereby the cooperative function may be executed. Now, a specificexample will be described with reference to FIG. 35. FIG. 35 illustratesan edit screen 196 that is displayed on the terminal apparatus 16. Onthe edit screen 196, for example, the device image 66 related to the PC(A), the device image 64 related to the MFP (B), the device image 138related to the projector (C), and a device image 198 related to a camera(D) are displayed. These images are images that have previously beenobtained and are stored in the memory of the server 14 or the terminalapparatus 16, as described above. These images are read from the memoryand displayed on the edit screen 196 when a cooperative function is tobe executed. The cooperative function is executed by linking orsuperimposing images on the edit screen 196, as described above. Withsuch a process being performed, it is not necessary to capture theimages of the devices that have previously been identified every time acooperative function is to be executed, and the cooperative function maybe efficiently executed. Furthermore, only the images of devicescaptured by the camera are displayed, and thus the visibility of thedevices to be operated may increase.

Each of the image forming apparatus 10, the terminal apparatuses 12 and16, and the servers 14 and 14A is implemented through cooperationbetween hardware resources and software resources, for example.Specifically, each of the image forming apparatus 10, the terminalapparatuses 12 and 16, and the servers 14 and 14A includes one or pluralprocessors, such as a central processing unit (CPU), which are notillustrated. The one or plural processors read and execute a programstored in a storage apparatus (not illustrated), and thereby thefunctions of the individual units of the image forming apparatus 10, theterminal apparatuses 12 and 16, and the servers 14 and 14A areimplemented. The program is stored in the storage apparatus through arecording medium, such as a compact disc (CD) or a digital versatiledisc (DVD), or through a communication path, such as a network.Alternatively, the individual units of the image forming apparatus 10,the terminal apparatuses 12 and 16, and the servers 14 and 14A may beimplemented by hardware resources such as a processor or an electroniccircuit. An apparatus such as a memory may be used for theimplementation. Alternatively, the individual units of the image formingapparatus 10, the terminal apparatuses 12 and 16, and the servers 14 and14A may be implemented by a digital signal processor (DSP) or a fieldprogrammable gate array (FPGA).

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A non-transitory computer readable medium storinga program that causes a computer to execute information processing, theinformation processing comprising: causing a display to display a firstfield of view of a camera directed to a first direction; capturing animage of a first target in the first direction; causing the display todisplay the captured image of the first target; and while continuing tocause the display to display the captured image of the first target,causing the display to display, behind the captured image of the firsttarget, a second field of view of the camera directed to a seconddirection that is different from the first direction.
 2. Thenon-transitory computer readable medium according to claim 1, whereinthe first target comprises a first device, and wherein the informationprocessing comprises: identifying the first device; identifying a seconddevice in the second direction; and sending an instruction to perform acooperative function using a first function of the first device and asecond function of the second device.
 3. The non-transitory computerreadable medium according to claim 1, wherein the first target comprisesa first device, and wherein the information processing comprises:identifying the first device; identifying a second device in the seconddirection; and prompting a user to confirm whether to cooperate thefirst device and the second device.
 4. The non-transitory computerreadable medium according to claim 1, wherein the first target comprisesa first device, and wherein the information processing comprises:identifying the first device; identifying a second device in the seconddirection; and causing the display to display a cooperative functionusing a first function of the first device and a second function of thesecond device.
 5. The non-transitory computer readable medium accordingto claim 1, wherein the information processing comprises: identifying asecond target in the second direction, the second target being differentfrom the first target and comprising a device; and sending aninstruction to the device to perform a function of the device.
 6. Thenon-transitory computer readable medium according to claim 1, whereinthe information processing comprises: identifying a second targetoverlapping the captured image, the second target being different fromthe first target and comprising a device located in the seconddirection; and sending an instruction to the device to perform afunction of the device.
 7. The non-transitory computer readable mediumaccording to claim 1, wherein the information processing comprises:identifying a second target in the second direction, the second targetbeing different from the first target and comprising a device; andsending an instruction to the device to display predeterminedinformation on a display of the identified device.
 8. The non-transitorycomputer readable medium according to claim 1, wherein the informationprocessing comprises: identifying a second target overlapping thecaptured image, the second target being different from the first targetand comprising a device located in the second direction; and sending aninstruction to the identified device to display predeterminedinformation on a display of the identified device.
 9. An informationprocessing apparatus comprising: a processor configured to: cause thedisplay to display a first field of view of a camera directed to a firstdirection; capture an image of a first target in the first direction;cause the display to display the captured image of the first target; andwhile continuing to cause the display to display the captured image ofthe first target, cause the display to display, behind the capturedimage of the first target, a second field of view of the camera directedto a second direction that is different from the first direction. 10.The information processing apparatus according to claim 9, wherein theinformation processing apparatus comprises the display and the camera,and wherein the information processing apparatus is mobile so that auser can direct the camera to the first direction and then to the seconddirection.
 11. A non-transitory computer readable medium storing aprogram that causes a computer to execute information processing, theinformation processing comprising: causing a display to display a fieldof view of a camera by causing the display to continuously display imagedata received from the camera as the image data is being received;capturing an image of a first target in the field of view of the cameraat a first time point; causing the display to display the captured imageof the first target; and continuing to cause the display to display,behind the captured image of the first target, the field of view of thecamera.
 12. The non-transitory computer readable medium according toclaim 11, wherein the first target comprises a first device, wherein theinformation processing comprises: identifying the first device;identifying a second device in the field of view at a second time pointthat is later than the first time point; and sending an instruction toperform a cooperative function using a first function of the firstdevice and a second function of the second device.
 13. Thenon-transitory computer readable medium according to claim 11, whereinthe first target comprises a first device, wherein the informationprocessing comprises: identifying the first device; identifying a seconddevice in the field of view at a second time point that is later thanthe first time point; and prompting a user to confirm whether tocooperate the first device and the second device.
 14. The non-transitorycomputer readable medium according to claim 11, wherein the first targetis a first device, wherein the information processing comprises:identifying the first device; identifying a second device in the fieldof view at a second time point that is later than the first time point;and causing the display to display a cooperative function using a firstfunction of the first device and a second function of the second device.15. The non-transitory computer readable medium according to claim 11,wherein the information processing comprises: identifying a secondtarget, the second target being different from the first target andcomprising a device in the field of view at a second time point that islater than the first time point; and sending an instruction to theidentified device to perform a function of the device.
 16. Thenon-transitory computer readable medium according to claim 11, whereinthe information processing comprises: identifying a second targetoverlapping the captured image, the second target being different fromthe first target and comprising a device in the field of view at asecond time point that is later than the first time point; and sendingan instruction to the identified device to perform a function of thedevice.
 17. The non-transitory computer readable medium according toclaim 11, wherein the information processing comprises: identifying asecond target, the second target being different from the first targetand comprising a device in the field of view at a second time point thatis later than the first time point; and sending an instruction to theidentified device to display predetermined information on a display ofthe identified device.
 18. The non-transitory computer readable mediumaccording to claim 11, wherein the information processing comprises:identifying a second target overlapping the captured image, the secondtarget being different from the first target and comprising a device inthe field of view at a second time point that is later than the firsttime point; and sending an instruction to the identified device todisplay predetermined information on a display of the identified device.